[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:34 dist-test-slave-qsm9 krb5kdc[474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:34 dist-test-slave-qsm9 krb5kdc[474](info): set up 2 sockets
Sep 26 20:20:34 dist-test-slave-qsm9 krb5kdc[474](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 20:20:35.055926 467 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:20:35 dist-test-slave-qsm9 krb5kdc[474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:20:35.191478 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:40663
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:41319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:40663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:35.306219 487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:35.306602 487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:35.306713 487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:35.310949 487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:35.311060 487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:35.311103 487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:35.311149 487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:35.311209 487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:35.316792 487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:41319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:40663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:40663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:35.317996 487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:35.319157 487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:35.328811 493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:35.328893 495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:35.328974 487 server_base.cc:1034] running on GCE node
W20260926 20:20:35.329064 492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:35.329790 487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:35.330948 487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:35.332110 487 hybrid_clock.cc:648] HybridClock initialized: now 1790454035332097 us; error 39 us; skew 500 ppm
Sep 26 20:20:35 dist-test-slave-qsm9 krb5kdc[474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:35.334983 487 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:20:35.336009 487 webserver.cc:460] Webserver started at http://127.0.116.254:43299/ using document root <none> and password file <none>
I20260926 20:20:35.336581 487 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:35.336673 487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:35.336915 487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:35.339001 487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "b6eaec9c9b4f4f3f9b8ab56f05dd6d2b"
format_stamp: "Formatted at 2026-09-26 20:20:35 on dist-test-slave-qsm9"
server_key: "67a614465058a037b86f5934669a50df"
server_key_iv: "33fbdb1b8a096705dcf219231c84dca3"
server_key_version: "encryptionkey@0"
I20260926 20:20:35.339629 487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "b6eaec9c9b4f4f3f9b8ab56f05dd6d2b"
format_stamp: "Formatted at 2026-09-26 20:20:35 on dist-test-slave-qsm9"
server_key: "67a614465058a037b86f5934669a50df"
server_key_iv: "33fbdb1b8a096705dcf219231c84dca3"
server_key_version: "encryptionkey@0"
I20260926 20:20:35.343698 487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:20:35.346439 502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:35.347841 487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:20:35.347990 487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "b6eaec9c9b4f4f3f9b8ab56f05dd6d2b"
format_stamp: "Formatted at 2026-09-26 20:20:35 on dist-test-slave-qsm9"
server_key: "67a614465058a037b86f5934669a50df"
server_key_iv: "33fbdb1b8a096705dcf219231c84dca3"
server_key_version: "encryptionkey@0"
I20260926 20:20:35.348186 487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:35.386909 487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:35.390125 487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:35.390357 487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:35.399236 487 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:40663
I20260926 20:20:35.400398 554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:40663 every 8 connection(s)
I20260926 20:20:35.401232 487 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:35.404722 555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:35.409088 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 487
I20260926 20:20:35.409247 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:20:35.410619 467 external_mini_cluster.cc:1428] Setting key 4d8c3e6c7a728a1d9245731e4cb07af5
I20260926 20:20:35.410962 555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b: Bootstrap starting.
I20260926 20:20:35.413928 555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:35.414877 555 log.cc:824] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:35.417174 555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b: No bootstrap required, opened a new log
I20260926 20:20:35.421730 555 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6eaec9c9b4f4f3f9b8ab56f05dd6d2b" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 40663 } }
I20260926 20:20:35.421974 555 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:35.422047 555 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6eaec9c9b4f4f3f9b8ab56f05dd6d2b, State: Initialized, Role: FOLLOWER
I20260926 20:20:35.422494 555 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6eaec9c9b4f4f3f9b8ab56f05dd6d2b" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 40663 } }
I20260926 20:20:35.422672 555 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:35.422742 555 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:35.422850 555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:35.424104 555 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6eaec9c9b4f4f3f9b8ab56f05dd6d2b" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 40663 } }
I20260926 20:20:35.424484 555 leader_election.cc:302] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6eaec9c9b4f4f3f9b8ab56f05dd6d2b; no voters:
I20260926 20:20:35.424816 555 leader_election.cc:288] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:35.424927 560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:35.425150 560 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b [term 1 LEADER]: Becoming Leader. State: Replica: b6eaec9c9b4f4f3f9b8ab56f05dd6d2b, State: Running, Role: LEADER
I20260926 20:20:35.425478 560 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6eaec9c9b4f4f3f9b8ab56f05dd6d2b" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 40663 } }
I20260926 20:20:35.426272 555 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:35.426715 562 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6eaec9c9b4f4f3f9b8ab56f05dd6d2b. Latest consensus state: current_term: 1 leader_uuid: "b6eaec9c9b4f4f3f9b8ab56f05dd6d2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6eaec9c9b4f4f3f9b8ab56f05dd6d2b" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 40663 } } }
I20260926 20:20:35.426836 562 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:35.427475 561 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6eaec9c9b4f4f3f9b8ab56f05dd6d2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6eaec9c9b4f4f3f9b8ab56f05dd6d2b" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 40663 } } }
I20260926 20:20:35.427623 561 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:35.427891 564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 20:20:35 dist-test-slave-qsm9 krb5kdc[474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:35.432096 564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:35.437958 564 catalog_manager.cc:1347] Generated new cluster ID: 0815cd9d49494294a357585c315392e9
I20260926 20:20:35.438036 564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:35.440867 557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:35.413828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57668 (local address 127.0.116.254:40663)
0926 20:20:35.414321 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:35.414339 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:35.415595 (+ 1256us) server_negotiation.cc:407] Connection header received
0926 20:20:35.419509 (+ 3914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:35.419546 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:35.419955 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:35.420330 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:35.422622 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:35.424165 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:35.425897 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:35.426605 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:35.426768 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:35.432613 (+ 5845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:35.432621 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:35.432638 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:35.432674 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:35.434964 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:35.435725 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:35.435732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:35.435738 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:35.435890 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:35.436333 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:35.436336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:35.436338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:35.436914 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:35.437149 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:35.439648 (+ 2499us) server_negotiation.cc:299] Negotiation successful
0926 20:20:35.439961 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":159,"threads_started":1}
I20260926 20:20:35.458824 564 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:35.460207 564 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:35.466799 564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6eaec9c9b4f4f3f9b8ab56f05dd6d2b: Generated new TSK 0
I20260926 20:20:35.467720 564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
I20260926 20:20:35.503787 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:40663
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:41319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:35.620009 581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:35.620301 581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:35.620347 581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:35.624590 581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:35.624661 581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:35.624758 581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:20:35.630712 581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:41319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:40663
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:35.631953 581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:35.633002 581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:35.643114 587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:35.643150 586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:35.643376 589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:35.643457 581 server_base.cc:1034] running on GCE node
I20260926 20:20:35.643970 581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:35.644697 581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:35.645891 581 hybrid_clock.cc:648] HybridClock initialized: now 1790454035645853 us; error 50 us; skew 500 ppm
Sep 26 20:20:35 dist-test-slave-qsm9 krb5kdc[474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:35.652969 581 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:20:35.654058 581 webserver.cc:460] Webserver started at http://127.0.116.193:39901/ using document root <none> and password file <none>
I20260926 20:20:35.654695 581 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:35.654793 581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:35.655038 581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:35.657290 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "c694c28dc0574292b4fb7a1a50e7762d"
format_stamp: "Formatted at 2026-09-26 20:20:35 on dist-test-slave-qsm9"
server_key: "076df70d3f140a81a5e0ec3f0611584d"
server_key_iv: "2d991d2a2baded0d8dab1d63467e65bd"
server_key_version: "encryptionkey@0"
I20260926 20:20:35.657898 581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "c694c28dc0574292b4fb7a1a50e7762d"
format_stamp: "Formatted at 2026-09-26 20:20:35 on dist-test-slave-qsm9"
server_key: "076df70d3f140a81a5e0ec3f0611584d"
server_key_iv: "2d991d2a2baded0d8dab1d63467e65bd"
server_key_version: "encryptionkey@0"
I20260926 20:20:35.662020 581 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260926 20:20:35.664794 596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:35.666096 581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:20:35.666253 581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "c694c28dc0574292b4fb7a1a50e7762d"
format_stamp: "Formatted at 2026-09-26 20:20:35 on dist-test-slave-qsm9"
server_key: "076df70d3f140a81a5e0ec3f0611584d"
server_key_iv: "2d991d2a2baded0d8dab1d63467e65bd"
server_key_version: "encryptionkey@0"
I20260926 20:20:35.666393 581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:35.682034 581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:35.685635 581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:35.685803 581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:35.686364 581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:20:35.687148 581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:35.687193 581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:35.687234 581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:35.687258 581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:35.696841 581 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:33987
I20260926 20:20:35.696863 709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:33987 every 8 connection(s)
I20260926 20:20:35.697893 581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:20:35.701042 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 581
I20260926 20:20:35.701198 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:20:35.701484 467 external_mini_cluster.cc:1428] Setting key 2d47dd27153e20ab8fcac6152c3b7267
Sep 26 20:20:35 dist-test-slave-qsm9 krb5kdc[474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.116.194@KRBTEST.COM; defaulting to no policy
I20260926 20:20:35.715188 557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:35.699983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:47727 (local address 127.0.116.254:40663)
0926 20:20:35.700140 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:35.700144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:35.701231 (+ 1087us) server_negotiation.cc:407] Connection header received
0926 20:20:35.702912 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:35.702915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:35.702967 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:35.703053 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:35.704943 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:35.705910 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:35.707208 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:35.707698 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:35.707824 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:35.710564 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:35.710571 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:35.710577 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:35.710614 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:35.712665 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:35.713345 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:35.713349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:35.713350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:35.713443 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:35.713855 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:35.713858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:35.713859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:35.714079 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:35.714193 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:35.714887 (+ 694us) server_negotiation.cc:299] Negotiation successful
0926 20:20:35.715041 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.194@KRBTEST.COM" created.
I20260926 20:20:35.716172 712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:35.700321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:40663 (local address 127.0.116.193:47727)
0926 20:20:35.701030 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:35.701093 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:35.702658 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:35.703384 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:35.703394 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:35.703966 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:35.704724 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:35.704747 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:35.706073 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:35.706075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:35.707065 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:35.707074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:35.707877 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:35.707880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:35.708035 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:35.708730 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:35.708748 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:35.710339 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:35.712850 (+ 2511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:35.712857 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:35.712874 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:35.713203 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:35.713546 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:35.713548 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:35.713549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:35.713737 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:35.714217 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:35.714224 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:35.714517 (+ 293us) client_negotiation.cc:769] Sending connection context
0926 20:20:35.714799 (+ 282us) client_negotiation.cc:240] Negotiation successful
0926 20:20:35.715128 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":115,"threads_started":1}
I20260926 20:20:35.717905 710 heartbeater.cc:347] Connected to a master server at 127.0.116.254:40663
I20260926 20:20:35.718238 710 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:35.718930 710 heartbeater.cc:511] Master 127.0.116.254:40663 requested a full tablet report, sending...
I20260926 20:20:35.720937 519 ts_manager.cc:194] Registered new tserver with Master: c694c28dc0574292b4fb7a1a50e7762d (127.0.116.193:33987)
I20260926 20:20:35.722129 519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:47727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
I20260926 20:20:35.735611 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:40663
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:41319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:35.881129 715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:35.881474 715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:35.881572 715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:35.885807 715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:35.885907 715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:35.886001 715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:20:35.892243 715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:41319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:40663
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:35.893594 715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:35.894658 715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:35.904865 720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:35.904855 723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:35.904842 721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:35.905260 715 server_base.cc:1034] running on GCE node
I20260926 20:20:35.905696 715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:35.906428 715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:35.907629 715 hybrid_clock.cc:648] HybridClock initialized: now 1790454035907553 us; error 52 us; skew 500 ppm
Sep 26 20:20:35 dist-test-slave-qsm9 krb5kdc[474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:35.911180 715 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:20:35.912221 715 webserver.cc:460] Webserver started at http://127.0.116.194:42251/ using document root <none> and password file <none>
I20260926 20:20:35.912881 715 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:35.912947 715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:35.913142 715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:35.915243 715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "5fcfb1b5f967419a9f082922cf8f10eb"
format_stamp: "Formatted at 2026-09-26 20:20:35 on dist-test-slave-qsm9"
server_key: "d14081c1748b952fdba2fe886d4d181d"
server_key_iv: "f83536afbd4f8b76e5a5674e131b521d"
server_key_version: "encryptionkey@0"
I20260926 20:20:35.915863 715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "5fcfb1b5f967419a9f082922cf8f10eb"
format_stamp: "Formatted at 2026-09-26 20:20:35 on dist-test-slave-qsm9"
server_key: "d14081c1748b952fdba2fe886d4d181d"
server_key_iv: "f83536afbd4f8b76e5a5674e131b521d"
server_key_version: "encryptionkey@0"
I20260926 20:20:35.919955 715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:20:35.922560 730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:35.924031 715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:20:35.924141 715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "5fcfb1b5f967419a9f082922cf8f10eb"
format_stamp: "Formatted at 2026-09-26 20:20:35 on dist-test-slave-qsm9"
server_key: "d14081c1748b952fdba2fe886d4d181d"
server_key_iv: "f83536afbd4f8b76e5a5674e131b521d"
server_key_version: "encryptionkey@0"
I20260926 20:20:35.924288 715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:35.946206 715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:35.949527 715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:35.949757 715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:35.950389 715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:20:35.951298 715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:35.951346 715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:35.951412 715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:35.951469 715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:35.960857 715 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:43571
I20260926 20:20:35.960881 843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:43571 every 8 connection(s)
I20260926 20:20:35.961851 715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:20:35.967731 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 715
I20260926 20:20:35.967860 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:20:35.968122 467 external_mini_cluster.cc:1428] Setting key fb6aabeb5ea1bf05f188d4a247673237
Sep 26 20:20:35 dist-test-slave-qsm9 krb5kdc[474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:35.977118 557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:35.963787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:41607 (local address 127.0.116.254:40663)
0926 20:20:35.963967 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:35.963972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:35.964828 (+ 856us) server_negotiation.cc:407] Connection header received
0926 20:20:35.966151 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:35.966153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:35.966213 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:35.966301 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:35.967675 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:35.968445 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:35.970015 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:35.970253 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:35.970351 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:35.973410 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:35.973423 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:35.973426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:35.973450 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:35.974963 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:35.975547 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:35.975553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:35.975555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:35.975644 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:35.975991 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:35.975994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:35.975996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:35.976198 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:35.976306 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:35.976828 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 20:20:35.976951 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
WARNING: no policy specified for kudu/127.0.116.195@KRBTEST.COM; defaulting to no policy
I20260926 20:20:35.977999 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:35.964158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:40663 (local address 127.0.116.194:41607)
0926 20:20:35.964651 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:35.964706 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:35.965916 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:35.966494 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:35.966501 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:35.966939 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:35.967428 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:35.967441 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:35.968599 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:35.968602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:35.969866 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:35.969881 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:35.970421 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:35.970428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:35.970666 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:35.971442 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:35.971458 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:35.973217 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:35.975088 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:35.975095 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:35.975112 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:35.975414 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:35.975739 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:35.975744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:35.975746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:35.975887 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:35.976303 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:35.976311 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:35.976576 (+ 265us) client_negotiation.cc:769] Sending connection context
0926 20:20:35.976774 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 20:20:35.977119 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":113,"threads_started":1}
I20260926 20:20:35.979306 844 heartbeater.cc:347] Connected to a master server at 127.0.116.254:40663
I20260926 20:20:35.979708 844 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:35.980317 844 heartbeater.cc:511] Master 127.0.116.254:40663 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.195@KRBTEST.COM" created.
I20260926 20:20:35.981684 519 ts_manager.cc:194] Registered new tserver with Master: 5fcfb1b5f967419a9f082922cf8f10eb (127.0.116.194:43571)
I20260926 20:20:35.982385 519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:41607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
I20260926 20:20:35.999332 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:40663
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:41319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:36.113878 849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:36.114173 849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:36.114219 849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:36.118266 849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:36.118325 849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:36.118383 849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:20:36.123461 849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:41319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:40663
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:36.124447 849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:36.125512 849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:36.135459 854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:36.135598 855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:36.135703 849 server_base.cc:1034] running on GCE node
W20260926 20:20:36.135454 857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:36.136240 849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:36.136950 849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:36.138162 849 hybrid_clock.cc:648] HybridClock initialized: now 1790454036138142 us; error 30 us; skew 500 ppm
Sep 26 20:20:36 dist-test-slave-qsm9 krb5kdc[474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:36.141461 849 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:20:36.142436 849 webserver.cc:460] Webserver started at http://127.0.116.195:36115/ using document root <none> and password file <none>
I20260926 20:20:36.143054 849 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:36.143148 849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:36.143405 849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:36.145581 849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "623924d6f23e458ea60023480d6b5ef0"
format_stamp: "Formatted at 2026-09-26 20:20:36 on dist-test-slave-qsm9"
server_key: "89695c19b6ce74f3ea5f87671bf31cbd"
server_key_iv: "242a2f427d7b30358808555f0468c349"
server_key_version: "encryptionkey@0"
I20260926 20:20:36.146143 849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "623924d6f23e458ea60023480d6b5ef0"
format_stamp: "Formatted at 2026-09-26 20:20:36 on dist-test-slave-qsm9"
server_key: "89695c19b6ce74f3ea5f87671bf31cbd"
server_key_iv: "242a2f427d7b30358808555f0468c349"
server_key_version: "encryptionkey@0"
I20260926 20:20:36.149804 849 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 20:20:36.152196 864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:36.153462 849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:20:36.153594 849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "623924d6f23e458ea60023480d6b5ef0"
format_stamp: "Formatted at 2026-09-26 20:20:36 on dist-test-slave-qsm9"
server_key: "89695c19b6ce74f3ea5f87671bf31cbd"
server_key_iv: "242a2f427d7b30358808555f0468c349"
server_key_version: "encryptionkey@0"
I20260926 20:20:36.153700 849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:36.183440 849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:36.186426 849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:36.186643 849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:36.187251 849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:20:36.188148 849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:36.188220 849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:36.188287 849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:36.188340 849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:36.197470 849 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:37919
I20260926 20:20:36.197490 977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:37919 every 8 connection(s)
I20260926 20:20:36.198450 849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:20:36.206166 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 849
I20260926 20:20:36.206354 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:20:36.206634 467 external_mini_cluster.cc:1428] Setting key a34376339ce45ed9c075ad4d31d93697
Sep 26 20:20:36 dist-test-slave-qsm9 krb5kdc[474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:36.212679 557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:36.200346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:46069 (local address 127.0.116.254:40663)
0926 20:20:36.200484 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:36.200487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:36.201264 (+ 777us) server_negotiation.cc:407] Connection header received
0926 20:20:36.202654 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:36.202657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:36.202721 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:36.202857 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:36.204175 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:36.204707 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:36.205863 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:36.206106 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:36.206219 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:36.209207 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:36.209211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:36.209214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:36.209244 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:36.210737 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:36.211204 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:36.211207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:36.211208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:36.211294 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:36.211663 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:36.211665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:36.211666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:36.211847 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:36.211923 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:36.212456 (+ 533us) server_negotiation.cc:299] Negotiation successful
0926 20:20:36.212557 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:20:36.213578 980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:36.200673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:40663 (local address 127.0.116.195:46069)
0926 20:20:36.201108 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:36.201150 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:36.202422 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:36.203047 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:36.203055 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:36.203442 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:36.204017 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:36.204031 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:36.204845 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:36.204848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:36.205734 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:36.205742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:36.206259 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:36.206271 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:36.206509 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:36.207165 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:36.207189 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:36.209012 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:36.210847 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:36.210852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:36.210865 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:36.211121 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:36.211372 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:36.211374 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:36.211376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:36.211574 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:36.211935 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:36.211940 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:36.212220 (+ 280us) client_negotiation.cc:769] Sending connection context
0926 20:20:36.212387 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 20:20:36.212645 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":107,"threads_started":1}
I20260926 20:20:36.214726 978 heartbeater.cc:347] Connected to a master server at 127.0.116.254:40663
I20260926 20:20:36.214952 978 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:36.215406 978 heartbeater.cc:511] Master 127.0.116.254:40663 requested a full tablet report, sending...
I20260926 20:20:36.216459 519 ts_manager.cc:194] Registered new tserver with Master: 623924d6f23e458ea60023480d6b5ef0 (127.0.116.195:37919)
I20260926 20:20:36.217104 519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:46069
I20260926 20:20:36.221347 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:20:36 dist-test-slave-qsm9 krb5kdc[474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454036, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:20:36 dist-test-slave-qsm9 krb5kdc[474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454036, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.116.193@KRBTEST.COM
I20260926 20:20:36.247743 985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:36.237708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.0.116.193:33987)
0926 20:20:36.238087 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:36.238093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:36.238119 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:20:36.238247 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:36.238252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:36.238450 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:36.238584 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:36.239288 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:36.240135 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:36.241086 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:36.241458 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:36.241548 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:36.244205 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:36.244216 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:36.244233 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:36.244257 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:36.245994 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:36.246445 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:36.246450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:36.246452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:36.246530 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:36.246825 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:36.246828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:36.246829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:36.247070 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:36.247238 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:36.247393 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 20:20:36.247585 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":109,"threads_started":1}
W20260926 20:20:36.248788 624 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48860
Sep 26 20:20:36 dist-test-slave-qsm9 krb5kdc[474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:20:36 dist-test-slave-qsm9 krb5kdc[474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.193@KRBTEST.COM
I20260926 20:20:36.274855 985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:36.266342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48874 (local address 127.0.116.193:33987)
0926 20:20:36.266530 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:36.266534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:36.266680 (+ 146us) server_negotiation.cc:407] Connection header received
0926 20:20:36.266854 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:36.266857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:36.266909 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:36.267000 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:36.267742 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:36.268251 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:36.269082 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:36.269311 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:36.269421 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:36.271938 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:36.271949 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:36.271951 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:36.271976 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:36.273239 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:36.273645 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:36.273649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:36.273651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:36.273731 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:36.274002 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:36.274007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:36.274008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:36.274178 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:36.274350 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:36.274554 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 20:20:36.274673 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:20:36.275949 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 581
I20260926 20:20:36.281535 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 715
I20260926 20:20:36.286763 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 849
I20260926 20:20:36.292011 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 487
2026-09-26T20:20:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2311 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:36 dist-test-slave-qsm9 krb5kdc[992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:36 dist-test-slave-qsm9 krb5kdc[992](info): set up 2 sockets
Sep 26 20:20:36 dist-test-slave-qsm9 krb5kdc[992](info): commencing operation
krb5kdc: starting...
W20260926 20:20:37.347455 467 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:20:37 dist-test-slave-qsm9 krb5kdc[992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:20:37.479223 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:44493
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:36757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:44493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:37.591041 1005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:37.591331 1005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:37.591373 1005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:37.595449 1005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:37.595527 1005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:37.595548 1005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:37.595564 1005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:37.595580 1005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:37.600920 1005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:36757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:44493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:44493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:37.601971 1005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:37.602977 1005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:37.611779 1013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:37.611783 1010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:37.611860 1005 server_base.cc:1034] running on GCE node
W20260926 20:20:37.611802 1011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:37.612651 1005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:37.613785 1005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:37.614975 1005 hybrid_clock.cc:648] HybridClock initialized: now 1790454037614937 us; error 46 us; skew 500 ppm
Sep 26 20:20:37 dist-test-slave-qsm9 krb5kdc[992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:37.617923 1005 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:20:37.618914 1005 webserver.cc:460] Webserver started at http://127.0.116.254:43303/ using document root <none> and password file <none>
I20260926 20:20:37.619554 1005 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:37.619620 1005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:37.619792 1005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:37.621798 1005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "9de25f052aac43cc817fe750b3a074ad"
format_stamp: "Formatted at 2026-09-26 20:20:37 on dist-test-slave-qsm9"
server_key: "c5e1a99a225b5c1d2042ea96bb1b6a3d"
server_key_iv: "32cfc7dc74dd3d1ca39fd5668d9a1ca2"
server_key_version: "encryptionkey@0"
I20260926 20:20:37.622331 1005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "9de25f052aac43cc817fe750b3a074ad"
format_stamp: "Formatted at 2026-09-26 20:20:37 on dist-test-slave-qsm9"
server_key: "c5e1a99a225b5c1d2042ea96bb1b6a3d"
server_key_iv: "32cfc7dc74dd3d1ca39fd5668d9a1ca2"
server_key_version: "encryptionkey@0"
I20260926 20:20:37.626053 1005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:20:37.628400 1020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:37.629719 1005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:20:37.629829 1005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "9de25f052aac43cc817fe750b3a074ad"
format_stamp: "Formatted at 2026-09-26 20:20:37 on dist-test-slave-qsm9"
server_key: "c5e1a99a225b5c1d2042ea96bb1b6a3d"
server_key_iv: "32cfc7dc74dd3d1ca39fd5668d9a1ca2"
server_key_version: "encryptionkey@0"
I20260926 20:20:37.629923 1005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:37.656919 1005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:37.660787 1005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:37.660969 1005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:37.668672 1005 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:44493
I20260926 20:20:37.668681 1072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:44493 every 8 connection(s)
I20260926 20:20:37.669718 1005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:37.672610 1073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:37.676080 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 1005
I20260926 20:20:37.676172 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:20:37.676437 467 external_mini_cluster.cc:1428] Setting key efcb83b0087176370a68c0bc91314017
I20260926 20:20:37.678472 1073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad: Bootstrap starting.
I20260926 20:20:37.681025 1073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:37.681836 1073 log.cc:824] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:37.683987 1073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad: No bootstrap required, opened a new log
Sep 26 20:20:37 dist-test-slave-qsm9 krb5kdc[992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:37.688040 1073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de25f052aac43cc817fe750b3a074ad" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44493 } }
I20260926 20:20:37.688340 1073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:37.688407 1073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9de25f052aac43cc817fe750b3a074ad, State: Initialized, Role: FOLLOWER
I20260926 20:20:37.688911 1073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de25f052aac43cc817fe750b3a074ad" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44493 } }
I20260926 20:20:37.689072 1073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:37.689148 1073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:37.689244 1073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:37.690408 1073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de25f052aac43cc817fe750b3a074ad" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44493 } }
I20260926 20:20:37.690780 1073 leader_election.cc:302] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9de25f052aac43cc817fe750b3a074ad; no voters:
I20260926 20:20:37.691124 1073 leader_election.cc:288] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:37.691457 1078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:37.691752 1078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad [term 1 LEADER]: Becoming Leader. State: Replica: 9de25f052aac43cc817fe750b3a074ad, State: Running, Role: LEADER
I20260926 20:20:37.692113 1078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de25f052aac43cc817fe750b3a074ad" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44493 } }
I20260926 20:20:37.692425 1073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:37.692624 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:37.677855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.0.116.254:44493)
0926 20:20:37.678289 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:37.678304 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:37.678343 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:20:37.679021 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:37.679039 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:37.679389 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:37.679748 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:37.680612 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:37.681681 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:37.682747 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:37.683215 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:37.683331 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:37.686200 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:37.686206 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:37.686223 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:37.686253 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:37.688985 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:37.689497 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:37.689504 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:37.689511 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:37.689643 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:37.689920 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:37.689924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:37.689925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:37.690456 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:37.690642 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:37.691381 (+ 739us) server_negotiation.cc:299] Negotiation successful
0926 20:20:37.691700 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":144,"threads_started":1}
I20260926 20:20:37.694226 1079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9de25f052aac43cc817fe750b3a074ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de25f052aac43cc817fe750b3a074ad" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44493 } } }
I20260926 20:20:37.694344 1079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:37.695295 1080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9de25f052aac43cc817fe750b3a074ad. Latest consensus state: current_term: 1 leader_uuid: "9de25f052aac43cc817fe750b3a074ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de25f052aac43cc817fe750b3a074ad" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44493 } } }
I20260926 20:20:37.695395 1080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad [sys.catalog]: This master's current role is: LEADER
W20260926 20:20:37.697209 1092 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:20:37.697289 1092 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:20:37.697355 1093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:37.698032 1093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:37.703364 1093 catalog_manager.cc:1347] Generated new cluster ID: 5e4765085edd4094bc77a5c9923863cc
I20260926 20:20:37.703436 1093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:37.722208 1093 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:37.723613 1093 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:37.742314 1093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9de25f052aac43cc817fe750b3a074ad: Generated new TSK 0
I20260926 20:20:37.743106 1093 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
I20260926 20:20:37.774362 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44493
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:36757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:37.887174 1099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:37.887437 1099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:37.887482 1099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:37.891443 1099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:37.891556 1099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:37.891669 1099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:20:37.897351 1099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:36757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44493
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:37.898432 1099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:37.899474 1099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:37.909351 1105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:37.909400 1099 server_base.cc:1034] running on GCE node
W20260926 20:20:37.909328 1104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:37.909353 1107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:37.910133 1099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:37.910768 1099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:37.911928 1099 hybrid_clock.cc:648] HybridClock initialized: now 1790454037911908 us; error 33 us; skew 500 ppm
Sep 26 20:20:37 dist-test-slave-qsm9 krb5kdc[992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:37.914826 1099 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:20:37.915773 1099 webserver.cc:460] Webserver started at http://127.0.116.193:42837/ using document root <none> and password file <none>
I20260926 20:20:37.916440 1099 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:37.916505 1099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:37.916675 1099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:37.918706 1099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "ba5e77ecde1a4a6c83a1e915c59d2a09"
format_stamp: "Formatted at 2026-09-26 20:20:37 on dist-test-slave-qsm9"
server_key: "01d3bb445b27b3fe217009560b4e5fcf"
server_key_iv: "14f4211c7bc69c31688657826422991f"
server_key_version: "encryptionkey@0"
I20260926 20:20:37.919224 1099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "ba5e77ecde1a4a6c83a1e915c59d2a09"
format_stamp: "Formatted at 2026-09-26 20:20:37 on dist-test-slave-qsm9"
server_key: "01d3bb445b27b3fe217009560b4e5fcf"
server_key_iv: "14f4211c7bc69c31688657826422991f"
server_key_version: "encryptionkey@0"
I20260926 20:20:37.922904 1099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 20:20:37.925318 1114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:37.926553 1099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 20:20:37.926669 1099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "ba5e77ecde1a4a6c83a1e915c59d2a09"
format_stamp: "Formatted at 2026-09-26 20:20:37 on dist-test-slave-qsm9"
server_key: "01d3bb445b27b3fe217009560b4e5fcf"
server_key_iv: "14f4211c7bc69c31688657826422991f"
server_key_version: "encryptionkey@0"
I20260926 20:20:37.926775 1099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:37.938632 1099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:37.941851 1099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:37.942045 1099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:37.942584 1099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:20:37.943384 1099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:37.943452 1099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:37.943538 1099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:37.943590 1099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:37.952123 1099 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:33587
I20260926 20:20:37.952195 1227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:33587 every 8 connection(s)
I20260926 20:20:37.953053 1099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:20:37.960817 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 1099
I20260926 20:20:37.960928 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:20:37.961212 467 external_mini_cluster.cc:1428] Setting key 2bf9916e710d99d40b5a237c216475e5
Sep 26 20:20:37 dist-test-slave-qsm9 krb5kdc[992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:37.968670 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:37.955079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:55367 (local address 127.0.116.254:44493)
0926 20:20:37.955247 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:37.955252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:37.956217 (+ 965us) server_negotiation.cc:407] Connection header received
0926 20:20:37.957658 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:37.957661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:37.957707 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:37.957778 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:37.958937 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:37.959459 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:37.960658 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:37.960944 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:37.961054 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:37.964298 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:37.964301 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:37.964303 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:37.964326 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:37.966083 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:37.966638 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:37.966645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:37.966650 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:37.966746 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:37.967098 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:37.967104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:37.967109 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:37.967368 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:37.967481 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:37.968259 (+ 778us) server_negotiation.cc:299] Negotiation successful
0926 20:20:37.968421 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 20:20:37.969445 1230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:37.955574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44493 (local address 127.0.116.193:55367)
0926 20:20:37.956040 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:37.956085 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:37.957453 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:37.957927 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:37.957934 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:37.958314 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:37.958808 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:37.958821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:37.959627 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:37.959631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:37.960512 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:37.960547 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:37.961115 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:37.961119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:37.961345 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:37.962107 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:37.962127 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:37.964127 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:37.966232 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:37.966237 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:37.966251 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:37.966534 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:37.966849 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:37.966851 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:37.966858 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:37.967001 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:37.967479 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:37.967485 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:37.967996 (+ 511us) client_negotiation.cc:769] Sending connection context
0926 20:20:37.968164 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 20:20:37.968445 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":110,"threads_started":1}
I20260926 20:20:37.970839 1228 heartbeater.cc:347] Connected to a master server at 127.0.116.254:44493
I20260926 20:20:37.971127 1228 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.116.194@KRBTEST.COM; defaulting to no policy
I20260926 20:20:37.971751 1228 heartbeater.cc:511] Master 127.0.116.254:44493 requested a full tablet report, sending...
I20260926 20:20:37.973383 1037 ts_manager.cc:194] Registered new tserver with Master: ba5e77ecde1a4a6c83a1e915c59d2a09 (127.0.116.193:33587)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.194@KRBTEST.COM" created.
I20260926 20:20:37.974582 1037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:55367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
I20260926 20:20:37.992511 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44493
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:36757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:38.103740 1233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:38.104058 1233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:38.104102 1233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:38.108507 1233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:38.108572 1233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:38.108634 1233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:20:38.114136 1233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:36757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44493
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:38.115293 1233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:38.116427 1233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:38.126019 1241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:38.126035 1238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:38.126235 1233 server_base.cc:1034] running on GCE node
W20260926 20:20:38.126045 1239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:38.126693 1233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:38.127415 1233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:38.128616 1233 hybrid_clock.cc:648] HybridClock initialized: now 1790454038128589 us; error 36 us; skew 500 ppm
Sep 26 20:20:38 dist-test-slave-qsm9 krb5kdc[992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:38.131808 1233 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:20:38.132750 1233 webserver.cc:460] Webserver started at http://127.0.116.194:38915/ using document root <none> and password file <none>
I20260926 20:20:38.133386 1233 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:38.133453 1233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:38.133787 1233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:38.136155 1233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "58993c52e153450c8a6d86a10f2a197e"
format_stamp: "Formatted at 2026-09-26 20:20:38 on dist-test-slave-qsm9"
server_key: "13d9e29c39ede4091000926d5819d824"
server_key_iv: "50e7977fcede6042137f99fd25c2bfaa"
server_key_version: "encryptionkey@0"
I20260926 20:20:38.136792 1233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "58993c52e153450c8a6d86a10f2a197e"
format_stamp: "Formatted at 2026-09-26 20:20:38 on dist-test-slave-qsm9"
server_key: "13d9e29c39ede4091000926d5819d824"
server_key_iv: "50e7977fcede6042137f99fd25c2bfaa"
server_key_version: "encryptionkey@0"
I20260926 20:20:38.141050 1233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:20:38.143898 1248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:38.145167 1233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:20:38.145298 1233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "58993c52e153450c8a6d86a10f2a197e"
format_stamp: "Formatted at 2026-09-26 20:20:38 on dist-test-slave-qsm9"
server_key: "13d9e29c39ede4091000926d5819d824"
server_key_iv: "50e7977fcede6042137f99fd25c2bfaa"
server_key_version: "encryptionkey@0"
I20260926 20:20:38.145408 1233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:38.165962 1233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:38.169994 1233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:38.170210 1233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:38.170855 1233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:20:38.171742 1233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:38.171790 1233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:38.171832 1233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:38.171890 1233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:38.180732 1233 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:40469
I20260926 20:20:38.180753 1361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:40469 every 8 connection(s)
I20260926 20:20:38.181679 1233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:20:38.189688 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 1233
I20260926 20:20:38.189842 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:20:38.190187 467 external_mini_cluster.cc:1428] Setting key 39f3c8b613c7ce233a2ab8477233f20e
Sep 26 20:20:38 dist-test-slave-qsm9 krb5kdc[992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:38.196529 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:38.183605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:46759 (local address 127.0.116.254:44493)
0926 20:20:38.183741 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:38.183745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:38.185131 (+ 1386us) server_negotiation.cc:407] Connection header received
0926 20:20:38.186039 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:38.186042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:38.186103 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:38.186205 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:38.187368 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.187959 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.189094 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.189327 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.189437 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:38.192163 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:38.192169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:38.192174 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:38.192206 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:38.193959 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:38.194474 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:38.194479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:38.194480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:38.194563 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:38.195001 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:38.195007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:38.195012 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:38.195296 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:38.195407 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:38.196169 (+ 762us) server_negotiation.cc:299] Negotiation successful
0926 20:20:38.196335 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:20:38.197645 1364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:38.184072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44493 (local address 127.0.116.194:46759)
0926 20:20:38.184969 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:38.185011 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:38.185828 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:38.186393 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:38.186400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:38.186753 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:38.187218 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:38.187230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.188080 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.188084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:38.188959 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:38.188967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.189474 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.189477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:38.189641 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:38.190295 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:38.190322 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:38.191996 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:38.194071 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:38.194079 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:38.194096 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:38.194371 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:38.194678 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:38.194681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:38.194682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:38.194888 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:38.195437 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:38.195445 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:38.195844 (+ 399us) client_negotiation.cc:769] Sending connection context
0926 20:20:38.196101 (+ 257us) client_negotiation.cc:240] Negotiation successful
0926 20:20:38.196409 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":550,"threads_started":1}
I20260926 20:20:38.199160 1362 heartbeater.cc:347] Connected to a master server at 127.0.116.254:44493
I20260926 20:20:38.199425 1362 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:38.199968 1362 heartbeater.cc:511] Master 127.0.116.254:44493 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.116.195@KRBTEST.COM; defaulting to no policy
I20260926 20:20:38.201051 1037 ts_manager.cc:194] Registered new tserver with Master: 58993c52e153450c8a6d86a10f2a197e (127.0.116.194:40469)
I20260926 20:20:38.201656 1037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:46759
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
I20260926 20:20:38.219770 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44493
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:36757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:38.332554 1367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:38.332840 1367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:38.332883 1367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:38.336987 1367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:38.337044 1367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:38.337102 1367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:20:38.342428 1367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:36757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44493
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:38.343593 1367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:38.344763 1367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:38.354918 1373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:38.354936 1375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:38.354938 1372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:38.355393 1367 server_base.cc:1034] running on GCE node
I20260926 20:20:38.355969 1367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:38.356645 1367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:38.357862 1367 hybrid_clock.cc:648] HybridClock initialized: now 1790454038357826 us; error 52 us; skew 500 ppm
Sep 26 20:20:38 dist-test-slave-qsm9 krb5kdc[992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:38.361150 1367 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:20:38.362229 1367 webserver.cc:460] Webserver started at http://127.0.116.195:43319/ using document root <none> and password file <none>
I20260926 20:20:38.362864 1367 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:38.362965 1367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:38.363219 1367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:38.365541 1367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "e71355c6e0c84189aece6a23e9806130"
format_stamp: "Formatted at 2026-09-26 20:20:38 on dist-test-slave-qsm9"
server_key: "898bd97e8d4898ce27276bc7918059e9"
server_key_iv: "a945b99333afd81d2fa25d94cd030c1a"
server_key_version: "encryptionkey@0"
I20260926 20:20:38.366179 1367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "e71355c6e0c84189aece6a23e9806130"
format_stamp: "Formatted at 2026-09-26 20:20:38 on dist-test-slave-qsm9"
server_key: "898bd97e8d4898ce27276bc7918059e9"
server_key_iv: "a945b99333afd81d2fa25d94cd030c1a"
server_key_version: "encryptionkey@0"
I20260926 20:20:38.370342 1367 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:20:38.372972 1382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:38.374264 1367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:20:38.374406 1367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "e71355c6e0c84189aece6a23e9806130"
format_stamp: "Formatted at 2026-09-26 20:20:38 on dist-test-slave-qsm9"
server_key: "898bd97e8d4898ce27276bc7918059e9"
server_key_iv: "a945b99333afd81d2fa25d94cd030c1a"
server_key_version: "encryptionkey@0"
I20260926 20:20:38.374528 1367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:38.396212 1367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:38.399282 1367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:38.399487 1367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:38.400094 1367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:20:38.400910 1367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:38.400985 1367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:38.401057 1367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:38.401108 1367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:38.410254 1367 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:42879
I20260926 20:20:38.410276 1495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:42879 every 8 connection(s)
I20260926 20:20:38.411190 1367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:20:38.416672 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 1367
I20260926 20:20:38.416967 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:20:38.417445 467 external_mini_cluster.cc:1428] Setting key a3a1f354a762b2e40d0d41edbbaa73c3
Sep 26 20:20:38 dist-test-slave-qsm9 krb5kdc[992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:38.426685 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:38.413035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:56119 (local address 127.0.116.254:44493)
0926 20:20:38.413170 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:38.413173 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:38.414049 (+ 876us) server_negotiation.cc:407] Connection header received
0926 20:20:38.415481 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:38.415485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:38.415581 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:38.415845 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:38.417324 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.418148 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.420246 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.420473 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.420553 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:38.423197 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:38.423200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:38.423202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:38.423227 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:38.424623 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:38.425170 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:38.425173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:38.425174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:38.425250 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:38.425611 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:38.425613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:38.425614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:38.425807 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:38.425881 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:38.426442 (+ 561us) server_negotiation.cc:299] Negotiation successful
0926 20:20:38.426559 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:20:38.427739 1498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:38.413440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44493 (local address 127.0.116.195:56119)
0926 20:20:38.413893 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:38.413937 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:38.415253 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:38.416041 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:38.416049 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:38.416486 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:38.417108 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:38.417134 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.418456 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.418463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:38.420099 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:38.420113 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.420644 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.420647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:38.420811 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:38.421487 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:38.421504 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:38.423032 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:38.424777 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:38.424782 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:38.424796 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:38.425072 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:38.425367 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:38.425370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:38.425372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:38.425523 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:38.425904 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:38.425909 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:38.426162 (+ 253us) client_negotiation.cc:769] Sending connection context
0926 20:20:38.426368 (+ 206us) client_negotiation.cc:240] Negotiation successful
0926 20:20:38.426648 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":120,"threads_started":1}
I20260926 20:20:38.428963 1496 heartbeater.cc:347] Connected to a master server at 127.0.116.254:44493
I20260926 20:20:38.429239 1496 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:38.429756 1496 heartbeater.cc:511] Master 127.0.116.254:44493 requested a full tablet report, sending...
I20260926 20:20:38.430998 1037 ts_manager.cc:194] Registered new tserver with Master: e71355c6e0c84189aece6a23e9806130 (127.0.116.195:42879)
I20260926 20:20:38.431844 1037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:56119
I20260926 20:20:38.432170 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:20:38.444372 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:38.436338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52910 (local address 127.0.116.254:44493)
0926 20:20:38.436520 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:38.436526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:38.436807 (+ 281us) server_negotiation.cc:407] Connection header received
0926 20:20:38.436903 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:38.436905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:38.436949 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:38.437024 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:38.437734 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.438269 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.439148 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.439395 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.439593 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:38.441156 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:38.441159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:38.441161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:38.441188 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:38.442561 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:38.442988 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:38.442990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:38.442992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:38.443064 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:38.443296 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:38.443298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:38.443299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:38.443607 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:38.443716 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:38.444031 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 20:20:38.444130 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:20:38.449559 1037 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:20:38.452737 1037 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:20:38.468940 1507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:38.462162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:33587 (local address 127.0.0.1:48550)
0926 20:20:38.463346 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:38.463410 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:38.463629 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:38.464264 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:38.464271 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:38.464297 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:38.464470 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:38.464478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.465868 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.465875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:38.467301 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:38.467312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.468296 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.468301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:38.468421 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:38.468463 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 20:20:38.468612 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 20:20:38.468750 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":29,"thread_start_us":116,"threads_started":1}
I20260926 20:20:38.468945 1508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:38.462268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48550 (local address 127.0.116.193:33587)
0926 20:20:38.463096 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:38.463111 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:38.463535 (+ 424us) server_negotiation.cc:407] Connection header received
0926 20:20:38.463875 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:38.463882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:38.464073 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:38.464269 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:38.464628 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.465717 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.467445 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.468149 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.468299 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:38.468474 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:38.468646 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 20:20:38.468754 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":108,"threads_started":1}
I20260926 20:20:38.469810 1511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:38.462310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:42879 (local address 127.0.0.1:43450)
0926 20:20:38.463426 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:38.463444 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:38.463657 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:38.464469 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:38.464472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:38.464491 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:38.464618 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:38.464625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.466008 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.466056 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:38.468318 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:38.468329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.469371 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.469374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:38.469476 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:38.469522 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 20:20:38.469593 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 20:20:38.469673 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"mutex_wait_us":1012,"thread_start_us":128,"threads_started":1}
I20260926 20:20:38.470243 1510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:38.462786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43450 (local address 127.0.116.195:42879)
0926 20:20:38.463261 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:38.463277 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:38.463573 (+ 296us) server_negotiation.cc:407] Connection header received
0926 20:20:38.463855 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:38.463862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:38.464091 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:38.464245 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:38.464726 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.465827 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.468461 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.469056 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.469174 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:38.469313 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:38.469976 (+ 663us) server_negotiation.cc:299] Negotiation successful
0926 20:20:38.470084 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":143,"threads_started":1}
I20260926 20:20:38.470402 1512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:38.463361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55318 (local address 127.0.116.194:40469)
0926 20:20:38.464000 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:38.464011 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:38.464556 (+ 545us) server_negotiation.cc:407] Connection header received
0926 20:20:38.464707 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:38.464712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:38.464897 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:38.465044 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:38.465343 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.466458 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.467546 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.468348 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.468430 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:38.468565 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:38.470095 (+ 1530us) server_negotiation.cc:299] Negotiation successful
0926 20:20:38.470225 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":96,"threads_started":1}
I20260926 20:20:38.470656 1509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:38.462958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:40469 (local address 127.0.0.1:55318)
0926 20:20:38.464449 (+ 1491us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:38.464461 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:38.464571 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:38.465060 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:38.465064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:38.465087 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:38.465210 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:38.465217 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.466591 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.466594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:38.467407 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:38.467416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.469625 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.469630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:38.469732 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:38.469760 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 20:20:38.470423 (+ 663us) client_negotiation.cc:240] Negotiation successful
0926 20:20:38.470530 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1438,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260926 20:20:38.472468 1162 tablet_service.cc:1467] Processing CreateTablet for tablet 379943f4318d4135817fdee300b7dc2c (DEFAULT_TABLE table=test-table [id=b43728787dbb4593bc218cd6eb143442]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:38.473627 1162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379943f4318d4135817fdee300b7dc2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:38.473376 1296 tablet_service.cc:1467] Processing CreateTablet for tablet 379943f4318d4135817fdee300b7dc2c (DEFAULT_TABLE table=test-table [id=b43728787dbb4593bc218cd6eb143442]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:38.473376 1430 tablet_service.cc:1467] Processing CreateTablet for tablet 379943f4318d4135817fdee300b7dc2c (DEFAULT_TABLE table=test-table [id=b43728787dbb4593bc218cd6eb143442]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:38.474447 1296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379943f4318d4135817fdee300b7dc2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:38.474447 1430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379943f4318d4135817fdee300b7dc2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:38.481149 1513 tablet_bootstrap.cc:492] T 379943f4318d4135817fdee300b7dc2c P ba5e77ecde1a4a6c83a1e915c59d2a09: Bootstrap starting.
I20260926 20:20:38.482455 1515 tablet_bootstrap.cc:492] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e: Bootstrap starting.
I20260926 20:20:38.482635 1514 tablet_bootstrap.cc:492] T 379943f4318d4135817fdee300b7dc2c P e71355c6e0c84189aece6a23e9806130: Bootstrap starting.
I20260926 20:20:38.483798 1513 tablet_bootstrap.cc:654] T 379943f4318d4135817fdee300b7dc2c P ba5e77ecde1a4a6c83a1e915c59d2a09: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:38.484619 1513 log.cc:824] T 379943f4318d4135817fdee300b7dc2c P ba5e77ecde1a4a6c83a1e915c59d2a09: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:38.484656 1514 tablet_bootstrap.cc:654] T 379943f4318d4135817fdee300b7dc2c P e71355c6e0c84189aece6a23e9806130: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:38.485163 1515 tablet_bootstrap.cc:654] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:38.485975 1514 log.cc:824] T 379943f4318d4135817fdee300b7dc2c P e71355c6e0c84189aece6a23e9806130: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:38.485975 1515 log.cc:824] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:38.487119 1513 tablet_bootstrap.cc:492] T 379943f4318d4135817fdee300b7dc2c P ba5e77ecde1a4a6c83a1e915c59d2a09: No bootstrap required, opened a new log
I20260926 20:20:38.487350 1513 ts_tablet_manager.cc:1397] T 379943f4318d4135817fdee300b7dc2c P ba5e77ecde1a4a6c83a1e915c59d2a09: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:20:38.487890 1515 tablet_bootstrap.cc:492] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e: No bootstrap required, opened a new log
I20260926 20:20:38.488082 1515 ts_tablet_manager.cc:1397] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:20:38.488556 1514 tablet_bootstrap.cc:492] T 379943f4318d4135817fdee300b7dc2c P e71355c6e0c84189aece6a23e9806130: No bootstrap required, opened a new log
I20260926 20:20:38.488729 1514 ts_tablet_manager.cc:1397] T 379943f4318d4135817fdee300b7dc2c P e71355c6e0c84189aece6a23e9806130: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:20:38.492384 1515 raft_consensus.cc:348] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e77ecde1a4a6c83a1e915c59d2a09" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33587 } } peers { permanent_uuid: "e71355c6e0c84189aece6a23e9806130" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 42879 } } peers { permanent_uuid: "58993c52e153450c8a6d86a10f2a197e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 40469 } }
I20260926 20:20:38.492370 1513 raft_consensus.cc:348] T 379943f4318d4135817fdee300b7dc2c P ba5e77ecde1a4a6c83a1e915c59d2a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e77ecde1a4a6c83a1e915c59d2a09" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33587 } } peers { permanent_uuid: "e71355c6e0c84189aece6a23e9806130" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 42879 } } peers { permanent_uuid: "58993c52e153450c8a6d86a10f2a197e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 40469 } }
I20260926 20:20:38.492733 1513 raft_consensus.cc:374] T 379943f4318d4135817fdee300b7dc2c P ba5e77ecde1a4a6c83a1e915c59d2a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:38.492719 1515 raft_consensus.cc:374] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:38.492799 1513 raft_consensus.cc:729] T 379943f4318d4135817fdee300b7dc2c P ba5e77ecde1a4a6c83a1e915c59d2a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba5e77ecde1a4a6c83a1e915c59d2a09, State: Initialized, Role: FOLLOWER
I20260926 20:20:38.492841 1515 raft_consensus.cc:729] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58993c52e153450c8a6d86a10f2a197e, State: Initialized, Role: FOLLOWER
I20260926 20:20:38.492772 1514 raft_consensus.cc:348] T 379943f4318d4135817fdee300b7dc2c P e71355c6e0c84189aece6a23e9806130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e77ecde1a4a6c83a1e915c59d2a09" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33587 } } peers { permanent_uuid: "e71355c6e0c84189aece6a23e9806130" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 42879 } } peers { permanent_uuid: "58993c52e153450c8a6d86a10f2a197e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 40469 } }
I20260926 20:20:38.492998 1514 raft_consensus.cc:374] T 379943f4318d4135817fdee300b7dc2c P e71355c6e0c84189aece6a23e9806130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:38.493045 1514 raft_consensus.cc:729] T 379943f4318d4135817fdee300b7dc2c P e71355c6e0c84189aece6a23e9806130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e71355c6e0c84189aece6a23e9806130, State: Initialized, Role: FOLLOWER
I20260926 20:20:38.493345 1513 consensus_queue.cc:260] T 379943f4318d4135817fdee300b7dc2c P ba5e77ecde1a4a6c83a1e915c59d2a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e77ecde1a4a6c83a1e915c59d2a09" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33587 } } peers { permanent_uuid: "e71355c6e0c84189aece6a23e9806130" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 42879 } } peers { permanent_uuid: "58993c52e153450c8a6d86a10f2a197e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 40469 } }
I20260926 20:20:38.493400 1514 consensus_queue.cc:260] T 379943f4318d4135817fdee300b7dc2c P e71355c6e0c84189aece6a23e9806130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e77ecde1a4a6c83a1e915c59d2a09" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33587 } } peers { permanent_uuid: "e71355c6e0c84189aece6a23e9806130" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 42879 } } peers { permanent_uuid: "58993c52e153450c8a6d86a10f2a197e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 40469 } }
I20260926 20:20:38.493400 1515 consensus_queue.cc:260] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e77ecde1a4a6c83a1e915c59d2a09" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33587 } } peers { permanent_uuid: "e71355c6e0c84189aece6a23e9806130" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 42879 } } peers { permanent_uuid: "58993c52e153450c8a6d86a10f2a197e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 40469 } }
I20260926 20:20:38.494316 1228 heartbeater.cc:503] Master 127.0.116.254:44493 was elected leader, sending a full tablet report...
I20260926 20:20:38.494462 1515 ts_tablet_manager.cc:1428] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260926 20:20:38.494462 1514 ts_tablet_manager.cc:1428] T 379943f4318d4135817fdee300b7dc2c P e71355c6e0c84189aece6a23e9806130: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:20:38.494539 1513 ts_tablet_manager.cc:1428] T 379943f4318d4135817fdee300b7dc2c P ba5e77ecde1a4a6c83a1e915c59d2a09: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 20:20:38.494782 1362 heartbeater.cc:503] Master 127.0.116.254:44493 was elected leader, sending a full tablet report...
I20260926 20:20:38.494779 1496 heartbeater.cc:503] Master 127.0.116.254:44493 was elected leader, sending a full tablet report...
I20260926 20:20:38.513525 1520 raft_consensus.cc:482] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:20:38.513705 1520 raft_consensus.cc:504] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e77ecde1a4a6c83a1e915c59d2a09" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33587 } } peers { permanent_uuid: "e71355c6e0c84189aece6a23e9806130" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 42879 } } peers { permanent_uuid: "58993c52e153450c8a6d86a10f2a197e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 40469 } }
I20260926 20:20:38.514842 1520 leader_election.cc:288] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba5e77ecde1a4a6c83a1e915c59d2a09 (127.0.116.193:33587), e71355c6e0c84189aece6a23e9806130 (127.0.116.195:42879)
I20260926 20:20:38.518903 1522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:38.515173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:42879 (local address 127.0.116.194:52091)
0926 20:20:38.515604 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:38.515615 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:38.515730 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:38.515990 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:38.515992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:38.516007 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:38.516132 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:38.516137 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.516838 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.516843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:38.517812 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:38.517821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.518473 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.518477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:38.518573 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:38.518601 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 20:20:38.518685 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:20:38.518752 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":48,"thread_start_us":152,"threads_started":1}
I20260926 20:20:38.518887 1510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:38.515342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:52091 (local address 127.0.116.195:42879)
0926 20:20:38.515475 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:38.515481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:38.515689 (+ 208us) server_negotiation.cc:407] Connection header received
0926 20:20:38.515837 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:38.515840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:38.515891 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:38.515978 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:38.516241 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.516718 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.517935 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.518351 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.518484 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:38.518538 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:38.518700 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 20:20:38.518776 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:20:38.519227 1364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:38.515052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:33587 (local address 127.0.116.194:44561)
0926 20:20:38.515200 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:38.515217 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:38.515324 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:38.515753 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:38.515756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:38.515777 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:38.515912 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:38.515919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.516830 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.516837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:38.517883 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:38.517893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.518716 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.518721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:38.518848 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:38.518867 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 20:20:38.518976 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 20:20:38.519075 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":31}
I20260926 20:20:38.519381 1508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:38.515245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:44561 (local address 127.0.116.193:33587)
0926 20:20:38.515423 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:38.515429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:38.515448 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:20:38.515538 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:38.515541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:38.515607 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:38.515726 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:38.516046 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.516702 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.518034 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.518554 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.518718 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:38.518777 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:38.519155 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 20:20:38.519233 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 20:20:38.519776 1450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379943f4318d4135817fdee300b7dc2c" candidate_uuid: "58993c52e153450c8a6d86a10f2a197e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e71355c6e0c84189aece6a23e9806130" is_pre_election: true
I20260926 20:20:38.520092 1450 raft_consensus.cc:2454] T 379943f4318d4135817fdee300b7dc2c P e71355c6e0c84189aece6a23e9806130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58993c52e153450c8a6d86a10f2a197e in term 0.
I20260926 20:20:38.520076 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379943f4318d4135817fdee300b7dc2c" candidate_uuid: "58993c52e153450c8a6d86a10f2a197e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba5e77ecde1a4a6c83a1e915c59d2a09" is_pre_election: true
I20260926 20:20:38.520380 1182 raft_consensus.cc:2454] T 379943f4318d4135817fdee300b7dc2c P ba5e77ecde1a4a6c83a1e915c59d2a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58993c52e153450c8a6d86a10f2a197e in term 0.
I20260926 20:20:38.520643 1250 leader_election.cc:302] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58993c52e153450c8a6d86a10f2a197e, e71355c6e0c84189aece6a23e9806130; no voters:
I20260926 20:20:38.521003 1520 raft_consensus.cc:2781] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:20:38.521082 1520 raft_consensus.cc:482] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:20:38.521114 1520 raft_consensus.cc:3037] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:38.522140 1520 raft_consensus.cc:504] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e77ecde1a4a6c83a1e915c59d2a09" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33587 } } peers { permanent_uuid: "e71355c6e0c84189aece6a23e9806130" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 42879 } } peers { permanent_uuid: "58993c52e153450c8a6d86a10f2a197e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 40469 } }
I20260926 20:20:38.522526 1520 leader_election.cc:288] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e [CANDIDATE]: Term 1 election: Requested vote from peers ba5e77ecde1a4a6c83a1e915c59d2a09 (127.0.116.193:33587), e71355c6e0c84189aece6a23e9806130 (127.0.116.195:42879)
I20260926 20:20:38.522821 1182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379943f4318d4135817fdee300b7dc2c" candidate_uuid: "58993c52e153450c8a6d86a10f2a197e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba5e77ecde1a4a6c83a1e915c59d2a09"
I20260926 20:20:38.522974 1182 raft_consensus.cc:3037] T 379943f4318d4135817fdee300b7dc2c P ba5e77ecde1a4a6c83a1e915c59d2a09 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:38.522938 1450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379943f4318d4135817fdee300b7dc2c" candidate_uuid: "58993c52e153450c8a6d86a10f2a197e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e71355c6e0c84189aece6a23e9806130"
I20260926 20:20:38.523152 1450 raft_consensus.cc:3037] T 379943f4318d4135817fdee300b7dc2c P e71355c6e0c84189aece6a23e9806130 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:38.524099 1182 raft_consensus.cc:2454] T 379943f4318d4135817fdee300b7dc2c P ba5e77ecde1a4a6c83a1e915c59d2a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58993c52e153450c8a6d86a10f2a197e in term 1.
I20260926 20:20:38.524430 1252 leader_election.cc:302] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58993c52e153450c8a6d86a10f2a197e, ba5e77ecde1a4a6c83a1e915c59d2a09; no voters:
I20260926 20:20:38.524484 1450 raft_consensus.cc:2454] T 379943f4318d4135817fdee300b7dc2c P e71355c6e0c84189aece6a23e9806130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58993c52e153450c8a6d86a10f2a197e in term 1.
I20260926 20:20:38.524621 1520 raft_consensus.cc:2781] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:38.524892 1520 raft_consensus.cc:686] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e [term 1 LEADER]: Becoming Leader. State: Replica: 58993c52e153450c8a6d86a10f2a197e, State: Running, Role: LEADER
I20260926 20:20:38.525241 1520 consensus_queue.cc:237] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e77ecde1a4a6c83a1e915c59d2a09" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33587 } } peers { permanent_uuid: "e71355c6e0c84189aece6a23e9806130" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 42879 } } peers { permanent_uuid: "58993c52e153450c8a6d86a10f2a197e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 40469 } }
I20260926 20:20:38.528574 1036 catalog_manager.cc:5495] T 379943f4318d4135817fdee300b7dc2c P 58993c52e153450c8a6d86a10f2a197e reported cstate change: term changed from 0 to 1, leader changed from <none> to 58993c52e153450c8a6d86a10f2a197e (127.0.116.194). New cstate: current_term: 1 leader_uuid: "58993c52e153450c8a6d86a10f2a197e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5e77ecde1a4a6c83a1e915c59d2a09" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e71355c6e0c84189aece6a23e9806130" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58993c52e153450c8a6d86a10f2a197e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 40469 } health_report { overall_health: HEALTHY } } }
Sep 26 20:20:38 dist-test-slave-qsm9 krb5kdc[992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.193@KRBTEST.COM
I20260926 20:20:38.552349 1508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:38.542985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.0.116.193:33587)
0926 20:20:38.543134 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:38.543137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:38.543256 (+ 119us) server_negotiation.cc:407] Connection header received
0926 20:20:38.543360 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:38.543362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:38.543401 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:38.543478 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:38.544276 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.544792 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.545597 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.545818 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.545907 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:38.548628 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:38.548641 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:38.548658 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:38.548684 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:38.550598 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:38.551015 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:38.551022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:38.551024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:38.551117 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:38.551380 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:38.551382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:38.551383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:38.551800 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:38.551955 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:38.552111 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 20:20:38.552209 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 20:20:38 dist-test-slave-qsm9 krb5kdc[992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454038, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:20:38 dist-test-slave-qsm9 krb5kdc[992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454038, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.116.193@KRBTEST.COM
I20260926 20:20:38.579996 1508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:38.571300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48568 (local address 127.0.116.193:33587)
0926 20:20:38.571475 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:38.571481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:38.571693 (+ 212us) server_negotiation.cc:407] Connection header received
0926 20:20:38.571885 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:38.571888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:38.571943 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:38.572032 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:38.572722 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.573259 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.574100 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.574338 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.574451 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:38.576955 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:38.576958 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:38.576960 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:38.576985 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:38.578450 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:38.578839 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:38.578849 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:38.578851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:38.578925 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:38.579187 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:38.579189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:38.579190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:38.579366 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:38.579463 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:38.579769 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 20:20:38.579875 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 20:20:38 dist-test-slave-qsm9 krb5kdc[992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:20:38 dist-test-slave-qsm9 krb5kdc[992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.193@KRBTEST.COM
I20260926 20:20:38.606189 1508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:38.597688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48572 (local address 127.0.116.193:33587)
0926 20:20:38.597881 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:38.597885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:38.597954 (+ 69us) server_negotiation.cc:407] Connection header received
0926 20:20:38.598097 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:38.598099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:38.598143 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:38.598205 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:38.598903 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.599397 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.600296 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:38.600502 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:38.600577 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:38.603030 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:38.603034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:38.603036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:38.603058 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:38.604511 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:38.605009 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:38.605012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:38.605013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:38.605089 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:38.605336 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:38.605338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:38.605339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:38.605499 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:38.605595 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:38.605866 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 20:20:38.605969 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260926 20:20:38.607606 1142 tablet_service.cc:2908] Rejecting scan request for tablet 379943f4318d4135817fdee300b7dc2c: Uninitialized: clean time has not yet been initialized
I20260926 20:20:38.609742 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 1099
I20260926 20:20:38.615262 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 1233
I20260926 20:20:38.620982 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 1367
I20260926 20:20:38.626566 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 1005
2026-09-26T20:20:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2334 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:38 dist-test-slave-qsm9 krb5kdc[1537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:38 dist-test-slave-qsm9 krb5kdc[1537](info): set up 2 sockets
Sep 26 20:20:38 dist-test-slave-qsm9 krb5kdc[1537](info): commencing operation
krb5kdc: starting...
W20260926 20:20:39.683619 467 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:20:39 dist-test-slave-qsm9 krb5kdc[1537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:20:39.814893 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:35875
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:35875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:39.930560 1550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:39.930864 1550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:39.930945 1550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:39.935096 1550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:39.935166 1550 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 20:20:39.935185 1550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:39.935204 1550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:39.935220 1550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:39.935282 1550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:39.940900 1550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:35875
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:35875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:39.942050 1550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:39.943228 1550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:39.952469 1555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:39.952458 1558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:39.952463 1556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:39.953172 1550 server_base.cc:1034] running on GCE node
I20260926 20:20:39.953913 1550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:39.955170 1550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:39.956367 1550 hybrid_clock.cc:648] HybridClock initialized: now 1790454039956319 us; error 67 us; skew 500 ppm
Sep 26 20:20:39 dist-test-slave-qsm9 krb5kdc[1537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:39.963689 1550 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:20:39.964833 1550 webserver.cc:460] Webserver started at http://127.0.116.254:34841/ using document root <none> and password file <none>
I20260926 20:20:39.965464 1550 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:39.965560 1550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:39.965808 1550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:39.968052 1550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "009abcca0dcb4e6db8081ef8c2af03c3"
format_stamp: "Formatted at 2026-09-26 20:20:39 on dist-test-slave-qsm9"
server_key: "00b713376f163ae59c236d6c678bec20"
server_key_iv: "332027b5cb8fb8ebac7f32b1bd3c5b54"
server_key_version: "encryptionkey@0"
I20260926 20:20:39.968645 1550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "009abcca0dcb4e6db8081ef8c2af03c3"
format_stamp: "Formatted at 2026-09-26 20:20:39 on dist-test-slave-qsm9"
server_key: "00b713376f163ae59c236d6c678bec20"
server_key_iv: "332027b5cb8fb8ebac7f32b1bd3c5b54"
server_key_version: "encryptionkey@0"
I20260926 20:20:39.972530 1550 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:20:39.974982 1565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:39.976500 1550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:20:39.976652 1550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "009abcca0dcb4e6db8081ef8c2af03c3"
format_stamp: "Formatted at 2026-09-26 20:20:39 on dist-test-slave-qsm9"
server_key: "00b713376f163ae59c236d6c678bec20"
server_key_iv: "332027b5cb8fb8ebac7f32b1bd3c5b54"
server_key_version: "encryptionkey@0"
I20260926 20:20:39.976778 1550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:39.991788 1550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:39.995817 1550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:39.996047 1550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:40.005484 1550 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:35875
I20260926 20:20:40.005509 1627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:35875 every 8 connection(s)
I20260926 20:20:40.006551 1550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:40.009584 1628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:40.011740 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 1550
I20260926 20:20:40.011864 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:20:40.012126 467 external_mini_cluster.cc:1428] Setting key 2a9d391d453c10cfb60947464da1c60a
I20260926 20:20:40.016559 1628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3: Bootstrap starting.
I20260926 20:20:40.018847 1628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:40.019676 1628 log.cc:824] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:20:40 dist-test-slave-qsm9 krb5kdc[1537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:40.022176 1628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3: No bootstrap required, opened a new log
I20260926 20:20:40.026255 1628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009abcca0dcb4e6db8081ef8c2af03c3" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 35875 } }
I20260926 20:20:40.026574 1628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:40.026655 1628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009abcca0dcb4e6db8081ef8c2af03c3, State: Initialized, Role: FOLLOWER
I20260926 20:20:40.027149 1628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009abcca0dcb4e6db8081ef8c2af03c3" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 35875 } }
I20260926 20:20:40.027300 1628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:40.027385 1628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:40.027482 1628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:40.027666 1631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:40.013739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48904 (local address 127.0.116.254:35875)
0926 20:20:40.014212 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:40.014228 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:40.014268 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:20:40.014926 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:40.014943 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:40.015283 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:40.015661 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:40.016446 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.017470 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.018325 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.018813 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.019002 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:40.022166 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:40.022172 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:40.022187 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:40.022221 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:40.024343 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:40.024847 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:40.024853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:40.024858 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:40.024979 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:40.025269 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:40.025272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:40.025273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:40.025710 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:40.025877 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:40.026173 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 20:20:40.026554 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":138,"threads_started":1}
I20260926 20:20:40.028474 1628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009abcca0dcb4e6db8081ef8c2af03c3" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 35875 } }
I20260926 20:20:40.028867 1628 leader_election.cc:302] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 009abcca0dcb4e6db8081ef8c2af03c3; no voters:
I20260926 20:20:40.029204 1628 leader_election.cc:288] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:40.029373 1633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:40.029652 1633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3 [term 1 LEADER]: Becoming Leader. State: Replica: 009abcca0dcb4e6db8081ef8c2af03c3, State: Running, Role: LEADER
I20260926 20:20:40.029997 1633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009abcca0dcb4e6db8081ef8c2af03c3" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 35875 } }
I20260926 20:20:40.030431 1628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:40.031675 1634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "009abcca0dcb4e6db8081ef8c2af03c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009abcca0dcb4e6db8081ef8c2af03c3" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 35875 } } }
I20260926 20:20:40.031919 1634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:40.032249 1634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 009abcca0dcb4e6db8081ef8c2af03c3. Latest consensus state: current_term: 1 leader_uuid: "009abcca0dcb4e6db8081ef8c2af03c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009abcca0dcb4e6db8081ef8c2af03c3" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 35875 } } }
I20260926 20:20:40.032405 1634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:40.032720 1645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:40.035759 1645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:40.041618 1645 catalog_manager.cc:1347] Generated new cluster ID: a482e75da12f4f32accb293be0b58317
I20260926 20:20:40.041695 1645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:40.076428 1645 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:40.077507 1645 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:40.084968 1645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3: Generated new TSK 0
I20260926 20:20:40.085724 1645 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
I20260926 20:20:40.119329 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:35875
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:40.243701 1654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:40.243995 1654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:40.244045 1654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:40.248787 1654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:40.248855 1654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:40.248898 1654 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:20:40.248972 1654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:20:40.254395 1654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.116.254:35875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:40.255515 1654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:40.256577 1654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:40.266381 1662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:40.266475 1654 server_base.cc:1034] running on GCE node
W20260926 20:20:40.266417 1660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:40.266613 1659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:40.267055 1654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:40.267776 1654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:40.268951 1654 hybrid_clock.cc:648] HybridClock initialized: now 1790454040268934 us; error 30 us; skew 500 ppm
Sep 26 20:20:40 dist-test-slave-qsm9 krb5kdc[1537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:40.271725 1654 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:20:40.272802 1654 webserver.cc:460] Webserver started at http://127.0.116.193:41541/ using document root <none> and password file <none>
I20260926 20:20:40.273350 1654 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:40.273408 1654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:40.273572 1654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:40.275627 1654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "bf9282b7099c4965952ce4c04f9c187a"
format_stamp: "Formatted at 2026-09-26 20:20:40 on dist-test-slave-qsm9"
server_key: "4804942ce8ebe32d8b999cd8c27dbdb7"
server_key_iv: "7208a5189b82a8e5239f820b7be1a0da"
server_key_version: "encryptionkey@0"
I20260926 20:20:40.276154 1654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "bf9282b7099c4965952ce4c04f9c187a"
format_stamp: "Formatted at 2026-09-26 20:20:40 on dist-test-slave-qsm9"
server_key: "4804942ce8ebe32d8b999cd8c27dbdb7"
server_key_iv: "7208a5189b82a8e5239f820b7be1a0da"
server_key_version: "encryptionkey@0"
I20260926 20:20:40.280050 1654 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:20:40.282644 1669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:40.284075 1654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:20:40.284214 1654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "bf9282b7099c4965952ce4c04f9c187a"
format_stamp: "Formatted at 2026-09-26 20:20:40 on dist-test-slave-qsm9"
server_key: "4804942ce8ebe32d8b999cd8c27dbdb7"
server_key_iv: "7208a5189b82a8e5239f820b7be1a0da"
server_key_version: "encryptionkey@0"
I20260926 20:20:40.284336 1654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:40.298079 1654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:40.301440 1654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:40.301663 1654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:40.303212 1654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:40.303283 1654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:40.303352 1654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:40.303395 1654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:20:40 dist-test-slave-qsm9 krb5kdc[1537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:40.314541 1654 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:44459
I20260926 20:20:40.314817 1784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:44459 every 8 connection(s)
I20260926 20:20:40.315693 1654 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:20:40.316540 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 1654
I20260926 20:20:40.316659 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:20:40.316956 467 external_mini_cluster.cc:1428] Setting key 622ebe06c2c1c907a1b3b6f2e857979d
I20260926 20:20:40.317924 1631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:40.304119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:44511 (local address 127.0.116.254:35875)
0926 20:20:40.304268 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:40.304271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:40.305351 (+ 1080us) server_negotiation.cc:407] Connection header received
0926 20:20:40.306403 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:40.306408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:40.306477 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:40.306538 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:40.307955 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.308902 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.310256 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.310584 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.310711 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:40.313752 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:40.313755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:40.313757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:40.313780 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:40.315448 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:40.316090 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:40.316094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:40.316097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:40.316206 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:40.316715 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:40.316719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:40.316721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:40.316952 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:40.317056 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:40.317650 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 20:20:40.317797 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:20:40.318809 1678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:40.304702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:35875 (local address 127.0.116.193:44511)
0926 20:20:40.305200 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:40.305242 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:40.306125 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:40.306880 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:40.306886 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:40.307221 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:40.307764 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:40.307778 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.309076 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.309080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:40.310103 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:40.310113 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.310837 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.310841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:40.310992 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:40.311673 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:40.311698 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:40.313545 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:40.315621 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:40.315626 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:40.315636 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:40.315963 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:40.316443 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:40.316449 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:40.316454 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:40.316620 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:40.317067 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:40.317073 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:40.317356 (+ 283us) client_negotiation.cc:769] Sending connection context
0926 20:20:40.317555 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 20:20:40.317888 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":141,"threads_started":1}
I20260926 20:20:40.320130 1785 heartbeater.cc:347] Connected to a master server at 127.0.116.254:35875
I20260926 20:20:40.320457 1785 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:40.321045 1785 heartbeater.cc:511] Master 127.0.116.254:35875 requested a full tablet report, sending...
I20260926 20:20:40.322726 1578 ts_manager.cc:194] Registered new tserver with Master: bf9282b7099c4965952ce4c04f9c187a (127.0.116.193:44459)
I20260926 20:20:40.324074 1578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:44511
WARNING: no policy specified for kudu/127.0.116.194@KRBTEST.COM; defaulting to no policy
I20260926 20:20:40.330806 1631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:40.322388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:55069 (local address 127.0.116.254:35875)
0926 20:20:40.322517 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:40.322522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:40.322731 (+ 209us) server_negotiation.cc:407] Connection header received
0926 20:20:40.322844 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:40.322849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:40.322909 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:40.323008 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:40.323528 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.324393 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.325823 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.326148 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.326267 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:40.327632 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:40.327636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:40.327640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:40.327668 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:40.329196 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:40.329610 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:40.329616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:40.329618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:40.329707 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:40.330054 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:40.330057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:40.330058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:40.330247 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:40.330323 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:40.330566 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 20:20:40.330696 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:20:40.330788 1792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:40.322333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:35875 (local address 127.0.116.193:55069)
0926 20:20:40.322634 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:40.322650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:40.322751 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:40.323035 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:40.323038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:40.323248 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:40.323365 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:40.323372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.324536 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.324539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:40.325681 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:40.325696 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.326287 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.326293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:40.326427 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:40.326932 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:40.326949 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:40.327433 (+ 484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:40.329302 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:40.329306 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:40.329309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:40.329496 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:40.329862 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:40.329866 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:40.329869 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:40.329966 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:40.330328 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:40.330330 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:40.330401 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 20:20:40.330478 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:20:40.330632 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":3,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
I20260926 20:20:40.346891 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:35875
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:40.461542 1794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:40.461895 1794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:40.461977 1794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:40.467947 1794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:40.468189 1794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:40.468271 1794 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:20:40.468361 1794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:20:40.474246 1794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.116.254:35875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:40.475453 1794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:40.476573 1794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:40.486377 1802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:40.486305 1799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:40.486503 1794 server_base.cc:1034] running on GCE node
W20260926 20:20:40.486315 1800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:40.487066 1794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:40.487883 1794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:40.489043 1794 hybrid_clock.cc:648] HybridClock initialized: now 1790454040489024 us; error 32 us; skew 500 ppm
Sep 26 20:20:40 dist-test-slave-qsm9 krb5kdc[1537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:40.491962 1794 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:20:40.492844 1794 webserver.cc:460] Webserver started at http://127.0.116.194:35577/ using document root <none> and password file <none>
I20260926 20:20:40.493392 1794 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:40.493453 1794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:40.493626 1794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:40.495837 1794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "4196f4ee33594a9d8d134440c9f000d4"
format_stamp: "Formatted at 2026-09-26 20:20:40 on dist-test-slave-qsm9"
server_key: "2ac97f0169c2c9f3b8d27b644736817d"
server_key_iv: "2e15c8aa8ae062931b9b9b82ff7a6c70"
server_key_version: "encryptionkey@0"
I20260926 20:20:40.496384 1794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "4196f4ee33594a9d8d134440c9f000d4"
format_stamp: "Formatted at 2026-09-26 20:20:40 on dist-test-slave-qsm9"
server_key: "2ac97f0169c2c9f3b8d27b644736817d"
server_key_iv: "2e15c8aa8ae062931b9b9b82ff7a6c70"
server_key_version: "encryptionkey@0"
I20260926 20:20:40.500303 1794 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 20:20:40.502844 1809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:40.504248 1794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:20:40.504395 1794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "4196f4ee33594a9d8d134440c9f000d4"
format_stamp: "Formatted at 2026-09-26 20:20:40 on dist-test-slave-qsm9"
server_key: "2ac97f0169c2c9f3b8d27b644736817d"
server_key_iv: "2e15c8aa8ae062931b9b9b82ff7a6c70"
server_key_version: "encryptionkey@0"
I20260926 20:20:40.504508 1794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:40.531903 1794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:40.535770 1794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:40.535992 1794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:40.537581 1794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:40.537644 1794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:40.537686 1794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:40.537748 1794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:40.548869 1794 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:45943
Sep 26 20:20:40 dist-test-slave-qsm9 krb5kdc[1537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:40.549191 1924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:45943 every 8 connection(s)
I20260926 20:20:40.550019 1794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:20:40.553978 1631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:40.538821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:53335 (local address 127.0.116.254:35875)
0926 20:20:40.538991 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:40.538996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:40.540163 (+ 1167us) server_negotiation.cc:407] Connection header received
0926 20:20:40.541660 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:40.541663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:40.541716 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:40.541797 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:40.543783 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.544383 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.546262 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.546495 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.546640 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:40.549894 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:40.549897 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:40.549899 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:40.549926 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:40.551833 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:40.552422 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:40.552435 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:40.552436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:40.552526 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:40.552902 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:40.552905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:40.552906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:40.553103 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:40.553185 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:40.553728 (+ 543us) server_negotiation.cc:299] Negotiation successful
0926 20:20:40.553851 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:20:40.553987 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 1794
I20260926 20:20:40.554100 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:20:40.554373 467 external_mini_cluster.cc:1428] Setting key 00e3552b43e8e3d992f8514e6d1cab57
I20260926 20:20:40.555135 1818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:40.539324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:35875 (local address 127.0.116.194:53335)
0926 20:20:40.539941 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:40.540013 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:40.541371 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:40.542041 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:40.542055 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:40.542644 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:40.543453 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:40.543478 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.544519 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.544524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:40.546107 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:40.546128 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.546678 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.546683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:40.546949 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:40.547765 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:40.547786 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:40.549683 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:40.551960 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:40.551965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:40.551978 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:40.552270 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:40.552627 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:40.552630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:40.552632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:40.552786 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:40.553199 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:40.553205 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:40.553468 (+ 263us) client_negotiation.cc:769] Sending connection context
0926 20:20:40.553626 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 20:20:40.554017 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":186,"threads_started":1}
I20260926 20:20:40.556484 1925 heartbeater.cc:347] Connected to a master server at 127.0.116.254:35875
I20260926 20:20:40.556666 1925 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:40.557199 1925 heartbeater.cc:511] Master 127.0.116.254:35875 requested a full tablet report, sending...
I20260926 20:20:40.558355 1582 ts_manager.cc:194] Registered new tserver with Master: 4196f4ee33594a9d8d134440c9f000d4 (127.0.116.194:45943)
I20260926 20:20:40.559175 1582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:53335
WARNING: no policy specified for kudu/127.0.116.195@KRBTEST.COM; defaulting to no policy
I20260926 20:20:40.567368 1631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:40.559023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:44807 (local address 127.0.116.254:35875)
0926 20:20:40.559143 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:40.559147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:40.559731 (+ 584us) server_negotiation.cc:407] Connection header received
0926 20:20:40.559867 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:40.559874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:40.559922 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:40.560009 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:40.560637 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.561420 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.562268 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.562503 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.562611 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:40.563874 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:40.563877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:40.563880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:40.563905 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:40.565506 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:40.565953 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:40.565956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:40.565959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:40.566045 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:40.566339 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:40.566343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:40.566345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:40.566683 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:40.566797 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:40.567060 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 20:20:40.567187 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:20:40.567384 1932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:40.558983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:35875 (local address 127.0.116.194:44807)
0926 20:20:40.559631 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:40.559648 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:40.559773 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:40.560051 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:40.560055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:40.560334 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:40.560495 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:40.560507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.561560 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.561563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:40.562132 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:40.562140 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.562616 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.562619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:40.562712 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:40.563286 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:40.563298 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:40.563724 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:40.565623 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:40.565627 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:40.565629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:40.565838 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:40.566137 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:40.566141 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:40.566143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:40.566243 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:40.566784 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:40.566787 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:40.566891 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 20:20:40.567002 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 20:20:40.567187 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
I20260926 20:20:40.584905 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:35875
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:40.698275 1934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:40.698596 1934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:40.698683 1934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:40.702850 1934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:40.702946 1934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:40.703023 1934 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:20:40.703105 1934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:20:40.708820 1934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.116.254:35875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:40.710044 1934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:40.711131 1934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:40.721395 1942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:40.721395 1940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:40.721434 1939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:40.721889 1934 server_base.cc:1034] running on GCE node
I20260926 20:20:40.722325 1934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:40.723117 1934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:40.724289 1934 hybrid_clock.cc:648] HybridClock initialized: now 1790454040724269 us; error 31 us; skew 500 ppm
Sep 26 20:20:40 dist-test-slave-qsm9 krb5kdc[1537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:40.727428 1934 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:20:40.728427 1934 webserver.cc:460] Webserver started at http://127.0.116.195:33429/ using document root <none> and password file <none>
I20260926 20:20:40.729045 1934 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:40.729141 1934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:40.729398 1934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:40.731990 1934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "b416c883c4a24b1f94adc7dfdb110f3a"
format_stamp: "Formatted at 2026-09-26 20:20:40 on dist-test-slave-qsm9"
server_key: "22449e2f0720a4cc981dbe687936ce32"
server_key_iv: "0640e9d801b032745d3541c56b391575"
server_key_version: "encryptionkey@0"
I20260926 20:20:40.732681 1934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "b416c883c4a24b1f94adc7dfdb110f3a"
format_stamp: "Formatted at 2026-09-26 20:20:40 on dist-test-slave-qsm9"
server_key: "22449e2f0720a4cc981dbe687936ce32"
server_key_iv: "0640e9d801b032745d3541c56b391575"
server_key_version: "encryptionkey@0"
I20260926 20:20:40.736742 1934 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:20:40.739550 1949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:40.741192 1934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:20:40.741365 1934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "b416c883c4a24b1f94adc7dfdb110f3a"
format_stamp: "Formatted at 2026-09-26 20:20:40 on dist-test-slave-qsm9"
server_key: "22449e2f0720a4cc981dbe687936ce32"
server_key_iv: "0640e9d801b032745d3541c56b391575"
server_key_version: "encryptionkey@0"
I20260926 20:20:40.741506 1934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:40.779461 1934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:40.783421 1934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:40.783689 1934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:40.785241 1934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:40.785322 1934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:40.785387 1934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:40.785437 1934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:20:40 dist-test-slave-qsm9 krb5kdc[1537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:40.797101 1934 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:38937
I20260926 20:20:40.797326 2064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:38937 every 8 connection(s)
I20260926 20:20:40.798274 1934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:20:40.800025 1631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:40.786306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:53391 (local address 127.0.116.254:35875)
0926 20:20:40.786458 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:40.786462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:40.787678 (+ 1216us) server_negotiation.cc:407] Connection header received
0926 20:20:40.789091 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:40.789097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:40.789174 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:40.789281 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:40.790622 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.791265 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.792575 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.792920 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.793044 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:40.795888 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:40.795893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:40.795896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:40.795928 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:40.797747 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:40.798296 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:40.798300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:40.798302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:40.798419 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:40.798784 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:40.798788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:40.798790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:40.799066 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:40.799170 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:40.799791 (+ 621us) server_negotiation.cc:299] Negotiation successful
0926 20:20:40.799907 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:20:40.800902 1958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:40.786948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:35875 (local address 127.0.116.195:53391)
0926 20:20:40.787462 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:40.787562 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:40.788749 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:40.789499 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:40.789512 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:40.789874 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:40.790463 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:40.790478 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.791393 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.791397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:40.792442 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:40.792451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.793093 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.793097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:40.793285 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:40.794017 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:40.794046 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:40.795662 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:40.797869 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:40.797874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:40.797886 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:40.798149 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:40.798530 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:40.798533 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:40.798534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:40.798668 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:40.799227 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:40.799234 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:40.799554 (+ 320us) client_negotiation.cc:769] Sending connection context
0926 20:20:40.799769 (+ 215us) client_negotiation.cc:240] Negotiation successful
0926 20:20:40.800036 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":148,"threads_started":1}
I20260926 20:20:40.802172 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 1934
I20260926 20:20:40.802304 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:20:40.802584 467 external_mini_cluster.cc:1428] Setting key 086eb4052d0a8ee6b2379442531ce418
I20260926 20:20:40.802878 2065 heartbeater.cc:347] Connected to a master server at 127.0.116.254:35875
I20260926 20:20:40.803042 2065 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:40.803722 2065 heartbeater.cc:511] Master 127.0.116.254:35875 requested a full tablet report, sending...
I20260926 20:20:40.804783 1578 ts_manager.cc:194] Registered new tserver with Master: b416c883c4a24b1f94adc7dfdb110f3a (127.0.116.195:38937)
I20260926 20:20:40.805542 1578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:53391
I20260926 20:20:40.811960 1631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:40.804990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:33739 (local address 127.0.116.254:35875)
0926 20:20:40.805119 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:40.805123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:40.805139 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:20:40.805182 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:40.805184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:40.805226 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:40.805308 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:40.805732 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.806405 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.807447 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.807751 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.807896 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:40.809018 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:40.809021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:40.809023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:40.809046 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:40.810540 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:40.810919 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:40.810922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:40.810923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:40.810994 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:40.811220 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:40.811222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:40.811223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:40.811382 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:40.811454 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:40.811748 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 20:20:40.811843 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:20:40.811971 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:40.804630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:35875 (local address 127.0.116.195:33739)
0926 20:20:40.805009 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:40.805026 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:40.805122 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:40.805337 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:40.805340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:40.805517 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:40.805614 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:40.805619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.806534 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.806538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:40.807329 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:40.807338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.807912 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.807916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:40.808010 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:40.808469 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:40.808481 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:40.808874 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:40.810635 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:40.810638 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:40.810639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:40.810827 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:40.811069 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:40.811071 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:40.811072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:40.811145 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:40.811458 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:40.811460 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:40.811597 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 20:20:40.811682 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 20:20:40.811794 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":132,"threads_started":1}
I20260926 20:20:40.816735 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:20:40 dist-test-slave-qsm9 krb5kdc[1537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454040, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:20:40 dist-test-slave-qsm9 krb5kdc[1537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454040, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:40.843168 1631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:40.834914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48912 (local address 127.0.116.254:35875)
0926 20:20:40.835064 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:40.835068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:40.835082 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:20:40.835125 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:40.835128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:40.835167 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:40.835249 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:40.836002 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.836526 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.837376 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.837601 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.837699 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:40.840264 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:40.840267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:40.840269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:40.840296 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:40.841753 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:40.842124 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:40.842126 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:40.842127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:40.842201 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:40.842435 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:40.842437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:40.842438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:40.842608 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:40.842706 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:40.842948 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 20:20:40.843045 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:20:40.846401 1578 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:20:40.849640 1578 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:20:40.865409 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:40.859423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:44459 (local address 127.0.0.1:55318)
0926 20:20:40.860310 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:40.860322 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:40.860398 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:40.860993 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:40.860999 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:40.861023 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:40.861213 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:40.861219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.862314 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.862320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:40.863822 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:40.863833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.864858 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.864862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:40.864976 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:40.865014 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 20:20:40.865123 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 20:20:40.865261 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":66,"spinlock_wait_cycles":1920,"thread_start_us":113,"threads_started":1}
I20260926 20:20:40.865409 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:40.859776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55318 (local address 127.0.116.193:44459)
0926 20:20:40.860051 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:40.860061 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:40.860490 (+ 429us) server_negotiation.cc:407] Connection header received
0926 20:20:40.860572 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:40.860578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:40.860843 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:40.861004 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:40.861330 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.862209 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.863943 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.864738 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.864858 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:40.865014 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:40.865158 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 20:20:40.865260 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":95,"threads_started":1}
I20260926 20:20:40.865770 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:40.859178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:38937 (local address 127.0.0.1:43226)
0926 20:20:40.860119 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:40.860172 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:40.860298 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:40.861020 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:40.861023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:40.861042 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:40.861224 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:40.861231 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.862626 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.862629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:40.864061 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:40.864068 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.865000 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.865003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:40.865091 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:40.865116 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:20:40.865166 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 20:20:40.865624 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":457,"thread_start_us":106,"threads_started":1}
I20260926 20:20:40.865970 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:40.859407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43226 (local address 127.0.116.195:38937)
0926 20:20:40.860586 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:40.860600 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:40.860625 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:20:40.860737 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:40.860740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:40.860885 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:40.861042 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:40.861362 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.862472 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.864168 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.864861 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.865255 (+ 394us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:40.865422 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:40.865471 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 20:20:40.865800 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1063,"thread_start_us":645,"threads_started":1}
I20260926 20:20:40.866400 2087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:40.859181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:45943 (local address 127.0.0.1:32774)
0926 20:20:40.861252 (+ 2071us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:40.861267 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:40.861353 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:40.861888 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:40.861890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:40.861909 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:40.862006 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:40.862011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.863311 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.863314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:40.864377 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:40.864387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.865985 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.865992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:40.866097 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:40.866122 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:20:40.866184 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 20:20:40.866252 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2012,"mutex_wait_us":1034,"thread_start_us":86,"threads_started":1}
I20260926 20:20:40.866991 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:40.859301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32774 (local address 127.0.116.194:45943)
0926 20:20:40.860588 (+ 1287us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:40.860601 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:40.861450 (+ 849us) server_negotiation.cc:407] Connection header received
0926 20:20:40.861541 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:40.861546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:40.861738 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:40.861888 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:40.862117 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.863182 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.864509 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:40.865314 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:40.865413 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:40.866718 (+ 1305us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:40.866771 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 20:20:40.866856 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1188,"thread_start_us":122,"threads_started":1}
I20260926 20:20:40.868597 1999 tablet_service.cc:1467] Processing CreateTablet for tablet ea69ad3f3891495482c7f7a8f687c606 (DEFAULT_TABLE table=test-table [id=3a42adb37ac7446dadcb9cc8a4228bae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:40.868659 1719 tablet_service.cc:1467] Processing CreateTablet for tablet ea69ad3f3891495482c7f7a8f687c606 (DEFAULT_TABLE table=test-table [id=3a42adb37ac7446dadcb9cc8a4228bae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:40.869504 1999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea69ad3f3891495482c7f7a8f687c606. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:40.869688 1719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea69ad3f3891495482c7f7a8f687c606. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:40.869359 1859 tablet_service.cc:1467] Processing CreateTablet for tablet ea69ad3f3891495482c7f7a8f687c606 (DEFAULT_TABLE table=test-table [id=3a42adb37ac7446dadcb9cc8a4228bae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:40.870263 1859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea69ad3f3891495482c7f7a8f687c606. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:40.876027 2088 tablet_bootstrap.cc:492] T ea69ad3f3891495482c7f7a8f687c606 P b416c883c4a24b1f94adc7dfdb110f3a: Bootstrap starting.
I20260926 20:20:40.877650 2089 tablet_bootstrap.cc:492] T ea69ad3f3891495482c7f7a8f687c606 P bf9282b7099c4965952ce4c04f9c187a: Bootstrap starting.
I20260926 20:20:40.878329 2088 tablet_bootstrap.cc:654] T ea69ad3f3891495482c7f7a8f687c606 P b416c883c4a24b1f94adc7dfdb110f3a: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:40.879187 2088 log.cc:824] T ea69ad3f3891495482c7f7a8f687c606 P b416c883c4a24b1f94adc7dfdb110f3a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:40.879690 2089 tablet_bootstrap.cc:654] T ea69ad3f3891495482c7f7a8f687c606 P bf9282b7099c4965952ce4c04f9c187a: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:40.879940 2090 tablet_bootstrap.cc:492] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4: Bootstrap starting.
I20260926 20:20:40.880503 2089 log.cc:824] T ea69ad3f3891495482c7f7a8f687c606 P bf9282b7099c4965952ce4c04f9c187a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:40.881469 2088 tablet_bootstrap.cc:492] T ea69ad3f3891495482c7f7a8f687c606 P b416c883c4a24b1f94adc7dfdb110f3a: No bootstrap required, opened a new log
I20260926 20:20:40.881644 2088 ts_tablet_manager.cc:1397] T ea69ad3f3891495482c7f7a8f687c606 P b416c883c4a24b1f94adc7dfdb110f3a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 20:20:40.881914 2090 tablet_bootstrap.cc:654] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:40.882277 2089 tablet_bootstrap.cc:492] T ea69ad3f3891495482c7f7a8f687c606 P bf9282b7099c4965952ce4c04f9c187a: No bootstrap required, opened a new log
I20260926 20:20:40.882447 2089 ts_tablet_manager.cc:1397] T ea69ad3f3891495482c7f7a8f687c606 P bf9282b7099c4965952ce4c04f9c187a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260926 20:20:40.882688 2090 log.cc:824] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:40.884239 2090 tablet_bootstrap.cc:492] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4: No bootstrap required, opened a new log
I20260926 20:20:40.884436 2090 ts_tablet_manager.cc:1397] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:20:40.885715 2088 raft_consensus.cc:348] T ea69ad3f3891495482c7f7a8f687c606 P b416c883c4a24b1f94adc7dfdb110f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } } peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } }
I20260926 20:20:40.885989 2088 raft_consensus.cc:374] T ea69ad3f3891495482c7f7a8f687c606 P b416c883c4a24b1f94adc7dfdb110f3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:40.886042 2088 raft_consensus.cc:729] T ea69ad3f3891495482c7f7a8f687c606 P b416c883c4a24b1f94adc7dfdb110f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b416c883c4a24b1f94adc7dfdb110f3a, State: Initialized, Role: FOLLOWER
I20260926 20:20:40.886500 2088 consensus_queue.cc:260] T ea69ad3f3891495482c7f7a8f687c606 P b416c883c4a24b1f94adc7dfdb110f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } } peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } }
I20260926 20:20:40.886911 2089 raft_consensus.cc:348] T ea69ad3f3891495482c7f7a8f687c606 P bf9282b7099c4965952ce4c04f9c187a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } } peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } }
I20260926 20:20:40.887145 2089 raft_consensus.cc:374] T ea69ad3f3891495482c7f7a8f687c606 P bf9282b7099c4965952ce4c04f9c187a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:40.887195 2089 raft_consensus.cc:729] T ea69ad3f3891495482c7f7a8f687c606 P bf9282b7099c4965952ce4c04f9c187a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9282b7099c4965952ce4c04f9c187a, State: Initialized, Role: FOLLOWER
I20260926 20:20:40.887506 2088 ts_tablet_manager.cc:1428] T ea69ad3f3891495482c7f7a8f687c606 P b416c883c4a24b1f94adc7dfdb110f3a: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260926 20:20:40.887481 2090 raft_consensus.cc:348] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } } peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } }
I20260926 20:20:40.887660 2089 consensus_queue.cc:260] T ea69ad3f3891495482c7f7a8f687c606 P bf9282b7099c4965952ce4c04f9c187a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } } peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } }
I20260926 20:20:40.887729 2090 raft_consensus.cc:374] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:40.887781 2090 raft_consensus.cc:729] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4196f4ee33594a9d8d134440c9f000d4, State: Initialized, Role: FOLLOWER
I20260926 20:20:40.887884 2065 heartbeater.cc:503] Master 127.0.116.254:35875 was elected leader, sending a full tablet report...
I20260926 20:20:40.888171 2090 consensus_queue.cc:260] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } } peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } }
I20260926 20:20:40.888449 2089 ts_tablet_manager.cc:1428] T ea69ad3f3891495482c7f7a8f687c606 P bf9282b7099c4965952ce4c04f9c187a: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260926 20:20:40.888763 1785 heartbeater.cc:503] Master 127.0.116.254:35875 was elected leader, sending a full tablet report...
I20260926 20:20:40.889146 2090 ts_tablet_manager.cc:1428] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260926 20:20:40.889487 1925 heartbeater.cc:503] Master 127.0.116.254:35875 was elected leader, sending a full tablet report...
I20260926 20:20:41.034828 2096 raft_consensus.cc:482] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:20:41.035077 2096 raft_consensus.cc:504] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } } peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } }
I20260926 20:20:41.036365 2096 leader_election.cc:288] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b416c883c4a24b1f94adc7dfdb110f3a (127.0.116.195:38937), bf9282b7099c4965952ce4c04f9c187a (127.0.116.193:44459)
I20260926 20:20:41.041268 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:41.037176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:50979 (local address 127.0.116.193:44459)
0926 20:20:41.037322 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:41.037325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:41.037336 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:20:41.037377 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:41.037379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:41.037413 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:41.037506 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:41.037755 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.038462 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.040133 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.040697 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.040848 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:41.040902 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:41.041050 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 20:20:41.041131 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:20:41.041332 2097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:41.036716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:44459 (local address 127.0.116.194:50979)
0926 20:20:41.037052 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:41.037067 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:41.037271 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:41.037541 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:41.037543 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:41.037558 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:41.037650 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:41.037654 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.038568 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.038572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:41.040008 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:41.040019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.040827 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.040830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:41.040918 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:41.040939 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 20:20:41.041010 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 20:20:41.041139 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":46,"spinlock_wait_cycles":87552,"thread_start_us":108,"threads_started":1}
I20260926 20:20:41.041603 1818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:41.036639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:38937 (local address 127.0.116.194:41929)
0926 20:20:41.036773 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:41.036791 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:41.036922 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:41.037186 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:41.037189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:41.037255 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:41.037397 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:41.037402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.038447 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.038450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:41.040279 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:41.040286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.041163 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.041167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:41.041268 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:41.041289 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 20:20:41.041367 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:20:41.041456 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":46,"spinlock_wait_cycles":24960}
I20260926 20:20:41.041749 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:41.036772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:41929 (local address 127.0.116.195:38937)
0926 20:20:41.036907 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:41.036911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:41.036928 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:20:41.037019 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:41.037022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:41.037072 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:41.037152 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:41.037506 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.038291 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.040387 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.040971 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.041107 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:41.041161 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:41.041545 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 20:20:41.041615 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:20:41.042140 1739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea69ad3f3891495482c7f7a8f687c606" candidate_uuid: "4196f4ee33594a9d8d134440c9f000d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9282b7099c4965952ce4c04f9c187a" is_pre_election: true
I20260926 20:20:41.042270 2019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea69ad3f3891495482c7f7a8f687c606" candidate_uuid: "4196f4ee33594a9d8d134440c9f000d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" is_pre_election: true
I20260926 20:20:41.042478 1739 raft_consensus.cc:2454] T ea69ad3f3891495482c7f7a8f687c606 P bf9282b7099c4965952ce4c04f9c187a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4196f4ee33594a9d8d134440c9f000d4 in term 0.
I20260926 20:20:41.042483 2019 raft_consensus.cc:2454] T ea69ad3f3891495482c7f7a8f687c606 P b416c883c4a24b1f94adc7dfdb110f3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4196f4ee33594a9d8d134440c9f000d4 in term 0.
I20260926 20:20:41.043016 1811 leader_election.cc:302] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4196f4ee33594a9d8d134440c9f000d4, bf9282b7099c4965952ce4c04f9c187a; no voters:
I20260926 20:20:41.043287 2096 raft_consensus.cc:2781] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:20:41.043368 2096 raft_consensus.cc:482] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:20:41.043412 2096 raft_consensus.cc:3037] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:41.044451 2096 raft_consensus.cc:504] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } } peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } }
I20260926 20:20:41.044837 2096 leader_election.cc:288] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4 [CANDIDATE]: Term 1 election: Requested vote from peers b416c883c4a24b1f94adc7dfdb110f3a (127.0.116.195:38937), bf9282b7099c4965952ce4c04f9c187a (127.0.116.193:44459)
I20260926 20:20:41.045225 2019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea69ad3f3891495482c7f7a8f687c606" candidate_uuid: "4196f4ee33594a9d8d134440c9f000d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b416c883c4a24b1f94adc7dfdb110f3a"
I20260926 20:20:41.045258 1739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea69ad3f3891495482c7f7a8f687c606" candidate_uuid: "4196f4ee33594a9d8d134440c9f000d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9282b7099c4965952ce4c04f9c187a"
I20260926 20:20:41.045333 2019 raft_consensus.cc:3037] T ea69ad3f3891495482c7f7a8f687c606 P b416c883c4a24b1f94adc7dfdb110f3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:41.045411 1739 raft_consensus.cc:3037] T ea69ad3f3891495482c7f7a8f687c606 P bf9282b7099c4965952ce4c04f9c187a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:41.046725 1739 raft_consensus.cc:2454] T ea69ad3f3891495482c7f7a8f687c606 P bf9282b7099c4965952ce4c04f9c187a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4196f4ee33594a9d8d134440c9f000d4 in term 1.
I20260926 20:20:41.046957 2019 raft_consensus.cc:2454] T ea69ad3f3891495482c7f7a8f687c606 P b416c883c4a24b1f94adc7dfdb110f3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4196f4ee33594a9d8d134440c9f000d4 in term 1.
I20260926 20:20:41.047061 1811 leader_election.cc:302] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4196f4ee33594a9d8d134440c9f000d4, bf9282b7099c4965952ce4c04f9c187a; no voters:
I20260926 20:20:41.047250 2096 raft_consensus.cc:2781] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:41.047551 2096 raft_consensus.cc:686] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4 [term 1 LEADER]: Becoming Leader. State: Replica: 4196f4ee33594a9d8d134440c9f000d4, State: Running, Role: LEADER
I20260926 20:20:41.047921 2096 consensus_queue.cc:237] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } } peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } }
W20260926 20:20:41.049413 2066 tablet.cc:2369] T ea69ad3f3891495482c7f7a8f687c606 P b416c883c4a24b1f94adc7dfdb110f3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:20:41.051285 1926 tablet.cc:2369] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:20:41.051261 1578 catalog_manager.cc:5495] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4196f4ee33594a9d8d134440c9f000d4 (127.0.116.194). New cstate: current_term: 1 leader_uuid: "4196f4ee33594a9d8d134440c9f000d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } health_report { overall_health: UNKNOWN } } }
W20260926 20:20:41.067049 1786 tablet.cc:2369] T ea69ad3f3891495482c7f7a8f687c606 P bf9282b7099c4965952ce4c04f9c187a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:20:41.106858 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:41.102704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32780 (local address 127.0.116.194:45943)
0926 20:20:41.102865 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:41.102870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:41.102975 (+ 105us) server_negotiation.cc:407] Connection header received
0926 20:20:41.103312 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:41.103315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:41.103372 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:41.103487 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:20:41.103809 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.104327 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.105256 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.105478 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.105591 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:41.105963 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:20:41.106425 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:20:41.106561 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:41.106641 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 20:20:41.106701 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 20:20:41.113191 1739 raft_consensus.cc:1261] T ea69ad3f3891495482c7f7a8f687c606 P bf9282b7099c4965952ce4c04f9c187a [term 1 FOLLOWER]: Refusing update from remote peer 4196f4ee33594a9d8d134440c9f000d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:41.113210 2019 raft_consensus.cc:1261] T ea69ad3f3891495482c7f7a8f687c606 P b416c883c4a24b1f94adc7dfdb110f3a [term 1 FOLLOWER]: Refusing update from remote peer 4196f4ee33594a9d8d134440c9f000d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:41.114077 2098 consensus_queue.cc:1035] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:20:41.114370 2096 consensus_queue.cc:1035] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:20:41.123732 2103 mvcc.cc:205] Tried to move back new op lower bound from 7333699752391421952 to 7333699752135331840. Current Snapshot: MvccSnapshot[applied={T|T < 7333699752391421952}]
I20260926 20:20:41.137006 1578 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:20:41.137213 1578 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:20:41.139891 1578 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3: Sending DeleteTablet for 3 replicas of tablet ea69ad3f3891495482c7f7a8f687c606
I20260926 20:20:41.140733 1999 tablet_service.cc:1514] Processing DeleteTablet for tablet ea69ad3f3891495482c7f7a8f687c606 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:41 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:43226
I20260926 20:20:41.140753 1859 tablet_service.cc:1514] Processing DeleteTablet for tablet ea69ad3f3891495482c7f7a8f687c606 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:41 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:32774
I20260926 20:20:41.141273 2114 tablet_replica.cc:330] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4: stopping tablet replica
I20260926 20:20:41.141536 1719 tablet_service.cc:1514] Processing DeleteTablet for tablet ea69ad3f3891495482c7f7a8f687c606 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:41 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:55318
I20260926 20:20:41.141633 2114 raft_consensus.cc:2229] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:20:41.142093 2114 raft_consensus.cc:2258] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:20:41.142304 2115 tablet_replica.cc:330] T ea69ad3f3891495482c7f7a8f687c606 P bf9282b7099c4965952ce4c04f9c187a: stopping tablet replica
I20260926 20:20:41.142499 2113 tablet_replica.cc:330] T ea69ad3f3891495482c7f7a8f687c606 P b416c883c4a24b1f94adc7dfdb110f3a: stopping tablet replica
I20260926 20:20:41.142612 2115 raft_consensus.cc:2229] T ea69ad3f3891495482c7f7a8f687c606 P bf9282b7099c4965952ce4c04f9c187a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:20:41.142653 1578 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 20:20:41.142776 2113 raft_consensus.cc:2229] T ea69ad3f3891495482c7f7a8f687c606 P b416c883c4a24b1f94adc7dfdb110f3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:20:41.142921 2115 raft_consensus.cc:2258] T ea69ad3f3891495482c7f7a8f687c606 P bf9282b7099c4965952ce4c04f9c187a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:20:41.143049 2113 raft_consensus.cc:2258] T ea69ad3f3891495482c7f7a8f687c606 P b416c883c4a24b1f94adc7dfdb110f3a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 20:20:41.143078 1578 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:20:41.144446 2115 ts_tablet_manager.cc:1905] T ea69ad3f3891495482c7f7a8f687c606 P bf9282b7099c4965952ce4c04f9c187a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:20:41.144996 2114 ts_tablet_manager.cc:1905] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:20:41.147641 2115 ts_tablet_manager.cc:1918] T ea69ad3f3891495482c7f7a8f687c606 P bf9282b7099c4965952ce4c04f9c187a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:20:41.147745 2115 log.cc:1196] T ea69ad3f3891495482c7f7a8f687c606 P bf9282b7099c4965952ce4c04f9c187a: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-0/wal/wals/ea69ad3f3891495482c7f7a8f687c606
I20260926 20:20:41.148065 2115 ts_tablet_manager.cc:1939] T ea69ad3f3891495482c7f7a8f687c606 P bf9282b7099c4965952ce4c04f9c187a: Deleting consensus metadata
I20260926 20:20:41.149135 2113 ts_tablet_manager.cc:1905] T ea69ad3f3891495482c7f7a8f687c606 P b416c883c4a24b1f94adc7dfdb110f3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:20:41.149619 1719 tablet_service.cc:1467] Processing CreateTablet for tablet 494bdf63e1114301aee16972a5c6f04b (DEFAULT_TABLE table=test-table [id=ce858237edc240b1b547a52e04bd552a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:41.149741 1859 tablet_service.cc:1467] Processing CreateTablet for tablet 494bdf63e1114301aee16972a5c6f04b (DEFAULT_TABLE table=test-table [id=ce858237edc240b1b547a52e04bd552a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:41.149873 1719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 494bdf63e1114301aee16972a5c6f04b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:41.149999 1859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 494bdf63e1114301aee16972a5c6f04b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:41.151942 2090 tablet_bootstrap.cc:492] T 494bdf63e1114301aee16972a5c6f04b P 4196f4ee33594a9d8d134440c9f000d4: Bootstrap starting.
I20260926 20:20:41.152029 2089 tablet_bootstrap.cc:492] T 494bdf63e1114301aee16972a5c6f04b P bf9282b7099c4965952ce4c04f9c187a: Bootstrap starting.
I20260926 20:20:41.152498 1567 catalog_manager.cc:4841] TS bf9282b7099c4965952ce4c04f9c187a (127.0.116.193:44459): tablet ea69ad3f3891495482c7f7a8f687c606 (table test-table [id=3a42adb37ac7446dadcb9cc8a4228bae]) successfully deleted
I20260926 20:20:41.152949 2090 tablet_bootstrap.cc:654] T 494bdf63e1114301aee16972a5c6f04b P 4196f4ee33594a9d8d134440c9f000d4: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:41.152994 2089 tablet_bootstrap.cc:654] T 494bdf63e1114301aee16972a5c6f04b P bf9282b7099c4965952ce4c04f9c187a: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:41.153028 2114 ts_tablet_manager.cc:1918] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:20:41.153159 2114 log.cc:1196] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/ea69ad3f3891495482c7f7a8f687c606
I20260926 20:20:41.149859 1999 tablet_service.cc:1467] Processing CreateTablet for tablet 494bdf63e1114301aee16972a5c6f04b (DEFAULT_TABLE table=test-table [id=ce858237edc240b1b547a52e04bd552a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:41.153465 1999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 494bdf63e1114301aee16972a5c6f04b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:41.154103 2114 ts_tablet_manager.cc:1939] T ea69ad3f3891495482c7f7a8f687c606 P 4196f4ee33594a9d8d134440c9f000d4: Deleting consensus metadata
I20260926 20:20:41.156184 1568 catalog_manager.cc:4841] TS 4196f4ee33594a9d8d134440c9f000d4 (127.0.116.194:45943): tablet ea69ad3f3891495482c7f7a8f687c606 (table test-table [id=3a42adb37ac7446dadcb9cc8a4228bae]) successfully deleted
I20260926 20:20:41.156399 2089 tablet_bootstrap.cc:492] T 494bdf63e1114301aee16972a5c6f04b P bf9282b7099c4965952ce4c04f9c187a: No bootstrap required, opened a new log
I20260926 20:20:41.156486 2089 ts_tablet_manager.cc:1397] T 494bdf63e1114301aee16972a5c6f04b P bf9282b7099c4965952ce4c04f9c187a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260926 20:20:41.156483 2088 tablet_bootstrap.cc:492] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a: Bootstrap starting.
I20260926 20:20:41.156556 2090 tablet_bootstrap.cc:492] T 494bdf63e1114301aee16972a5c6f04b P 4196f4ee33594a9d8d134440c9f000d4: No bootstrap required, opened a new log
I20260926 20:20:41.156634 2090 ts_tablet_manager.cc:1397] T 494bdf63e1114301aee16972a5c6f04b P 4196f4ee33594a9d8d134440c9f000d4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260926 20:20:41.157070 2090 raft_consensus.cc:348] T 494bdf63e1114301aee16972a5c6f04b P 4196f4ee33594a9d8d134440c9f000d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } }
I20260926 20:20:41.157202 2090 raft_consensus.cc:374] T 494bdf63e1114301aee16972a5c6f04b P 4196f4ee33594a9d8d134440c9f000d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:41.157253 2090 raft_consensus.cc:729] T 494bdf63e1114301aee16972a5c6f04b P 4196f4ee33594a9d8d134440c9f000d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4196f4ee33594a9d8d134440c9f000d4, State: Initialized, Role: FOLLOWER
I20260926 20:20:41.157374 2089 raft_consensus.cc:348] T 494bdf63e1114301aee16972a5c6f04b P bf9282b7099c4965952ce4c04f9c187a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } }
I20260926 20:20:41.157383 2090 consensus_queue.cc:260] T 494bdf63e1114301aee16972a5c6f04b P 4196f4ee33594a9d8d134440c9f000d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } }
I20260926 20:20:41.157477 2089 raft_consensus.cc:374] T 494bdf63e1114301aee16972a5c6f04b P bf9282b7099c4965952ce4c04f9c187a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:41.157509 2089 raft_consensus.cc:729] T 494bdf63e1114301aee16972a5c6f04b P bf9282b7099c4965952ce4c04f9c187a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9282b7099c4965952ce4c04f9c187a, State: Initialized, Role: FOLLOWER
I20260926 20:20:41.157552 2088 tablet_bootstrap.cc:654] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:41.157624 2089 consensus_queue.cc:260] T 494bdf63e1114301aee16972a5c6f04b P bf9282b7099c4965952ce4c04f9c187a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } }
I20260926 20:20:41.157677 2090 ts_tablet_manager.cc:1428] T 494bdf63e1114301aee16972a5c6f04b P 4196f4ee33594a9d8d134440c9f000d4: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260926 20:20:41.157810 2113 ts_tablet_manager.cc:1918] T ea69ad3f3891495482c7f7a8f687c606 P b416c883c4a24b1f94adc7dfdb110f3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:20:41.157871 2089 ts_tablet_manager.cc:1428] T 494bdf63e1114301aee16972a5c6f04b P bf9282b7099c4965952ce4c04f9c187a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260926 20:20:41.157898 2113 log.cc:1196] T ea69ad3f3891495482c7f7a8f687c606 P b416c883c4a24b1f94adc7dfdb110f3a: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/ea69ad3f3891495482c7f7a8f687c606
I20260926 20:20:41.158203 2113 ts_tablet_manager.cc:1939] T ea69ad3f3891495482c7f7a8f687c606 P b416c883c4a24b1f94adc7dfdb110f3a: Deleting consensus metadata
I20260926 20:20:41.159758 1569 catalog_manager.cc:4841] TS b416c883c4a24b1f94adc7dfdb110f3a (127.0.116.195:38937): tablet ea69ad3f3891495482c7f7a8f687c606 (table test-table [id=3a42adb37ac7446dadcb9cc8a4228bae]) successfully deleted
I20260926 20:20:41.160732 2088 tablet_bootstrap.cc:492] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a: No bootstrap required, opened a new log
I20260926 20:20:41.160841 2088 ts_tablet_manager.cc:1397] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 20:20:41.161357 2088 raft_consensus.cc:348] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } }
I20260926 20:20:41.161456 2088 raft_consensus.cc:374] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:41.161478 2088 raft_consensus.cc:729] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b416c883c4a24b1f94adc7dfdb110f3a, State: Initialized, Role: FOLLOWER
I20260926 20:20:41.161625 2088 consensus_queue.cc:260] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } }
I20260926 20:20:41.161886 2088 ts_tablet_manager.cc:1428] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 20:20:41.381470 2094 raft_consensus.cc:482] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:20:41.381632 2094 raft_consensus.cc:504] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } }
I20260926 20:20:41.382840 2094 leader_election.cc:288] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9282b7099c4965952ce4c04f9c187a (127.0.116.193:44459), 4196f4ee33594a9d8d134440c9f000d4 (127.0.116.194:45943)
I20260926 20:20:41.386842 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:41.383257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:53053 (local address 127.0.116.194:45943)
0926 20:20:41.383380 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:41.383384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:41.383611 (+ 227us) server_negotiation.cc:407] Connection header received
0926 20:20:41.383790 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:41.383793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:41.383876 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:41.383954 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:41.384221 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.384697 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.385863 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.386253 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.386344 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:41.386426 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:41.386630 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 20:20:41.386700 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:20:41.386850 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:41.383180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:53889 (local address 127.0.116.193:44459)
0926 20:20:41.383327 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:41.383332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:41.383532 (+ 200us) server_negotiation.cc:407] Connection header received
0926 20:20:41.383709 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:41.383714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:41.383771 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:41.383847 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:41.384206 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.384729 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.385896 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.386290 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.386438 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:41.386488 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:41.386669 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 20:20:41.386743 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 20:20:41.386857 2119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:41.383053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:44459 (local address 127.0.116.195:53889)
0926 20:20:41.383393 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:41.383409 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:41.383574 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:41.383879 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:41.383882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:41.383900 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:41.384064 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:41.384071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.384867 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.384869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:41.385761 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:41.385769 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.386428 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.386432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:41.386533 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:41.386549 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 20:20:41.386641 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 20:20:41.386707 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":74,"thread_start_us":86,"threads_started":1}
I20260926 20:20:41.386860 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:41.383182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:45943 (local address 127.0.116.195:53053)
0926 20:20:41.383483 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:41.383541 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:41.383682 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:41.383984 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:41.383986 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:41.384000 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:41.384102 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:41.384107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.384802 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.384808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:41.385741 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:41.385750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.386384 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.386391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:41.386495 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:41.386523 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 20:20:41.386593 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 20:20:41.386694 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":49,"thread_start_us":64,"threads_started":1}
I20260926 20:20:41.387342 1739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "494bdf63e1114301aee16972a5c6f04b" candidate_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9282b7099c4965952ce4c04f9c187a" is_pre_election: true
I20260926 20:20:41.387568 1739 raft_consensus.cc:2454] T 494bdf63e1114301aee16972a5c6f04b P bf9282b7099c4965952ce4c04f9c187a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b416c883c4a24b1f94adc7dfdb110f3a in term 0.
I20260926 20:20:41.387717 1879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "494bdf63e1114301aee16972a5c6f04b" candidate_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4196f4ee33594a9d8d134440c9f000d4" is_pre_election: true
I20260926 20:20:41.387945 1879 raft_consensus.cc:2454] T 494bdf63e1114301aee16972a5c6f04b P 4196f4ee33594a9d8d134440c9f000d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b416c883c4a24b1f94adc7dfdb110f3a in term 0.
I20260926 20:20:41.387928 1951 leader_election.cc:302] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b416c883c4a24b1f94adc7dfdb110f3a, bf9282b7099c4965952ce4c04f9c187a; no voters:
I20260926 20:20:41.388195 2094 raft_consensus.cc:2781] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:20:41.388274 2094 raft_consensus.cc:482] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:20:41.388324 2094 raft_consensus.cc:3037] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:41.389467 2094 raft_consensus.cc:504] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } }
I20260926 20:20:41.389802 2094 leader_election.cc:288] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a [CANDIDATE]: Term 1 election: Requested vote from peers bf9282b7099c4965952ce4c04f9c187a (127.0.116.193:44459), 4196f4ee33594a9d8d134440c9f000d4 (127.0.116.194:45943)
I20260926 20:20:41.390105 1739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "494bdf63e1114301aee16972a5c6f04b" candidate_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9282b7099c4965952ce4c04f9c187a"
I20260926 20:20:41.390141 1879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "494bdf63e1114301aee16972a5c6f04b" candidate_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4196f4ee33594a9d8d134440c9f000d4"
I20260926 20:20:41.390230 1739 raft_consensus.cc:3037] T 494bdf63e1114301aee16972a5c6f04b P bf9282b7099c4965952ce4c04f9c187a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:41.390288 1879 raft_consensus.cc:3037] T 494bdf63e1114301aee16972a5c6f04b P 4196f4ee33594a9d8d134440c9f000d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:41.391141 1739 raft_consensus.cc:2454] T 494bdf63e1114301aee16972a5c6f04b P bf9282b7099c4965952ce4c04f9c187a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b416c883c4a24b1f94adc7dfdb110f3a in term 1.
I20260926 20:20:41.391201 1879 raft_consensus.cc:2454] T 494bdf63e1114301aee16972a5c6f04b P 4196f4ee33594a9d8d134440c9f000d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b416c883c4a24b1f94adc7dfdb110f3a in term 1.
I20260926 20:20:41.391429 1951 leader_election.cc:302] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b416c883c4a24b1f94adc7dfdb110f3a, bf9282b7099c4965952ce4c04f9c187a; no voters:
I20260926 20:20:41.391640 2094 raft_consensus.cc:2781] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:41.391835 2094 raft_consensus.cc:686] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a [term 1 LEADER]: Becoming Leader. State: Replica: b416c883c4a24b1f94adc7dfdb110f3a, State: Running, Role: LEADER
I20260926 20:20:41.392095 2094 consensus_queue.cc:237] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } }
I20260926 20:20:41.394698 1581 catalog_manager.cc:5495] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a reported cstate change: term changed from 0 to 1, leader changed from <none> to b416c883c4a24b1f94adc7dfdb110f3a (127.0.116.195). New cstate: current_term: 1 leader_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } health_report { overall_health: HEALTHY } } }
Sep 26 20:20:41 dist-test-slave-qsm9 krb5kdc[1537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:41.483165 2129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:41.473716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55800 (local address 127.0.116.254:35875)
0926 20:20:41.474015 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:41.474022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:41.474037 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:20:41.474229 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:41.474268 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:41.474335 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:41.474465 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:41.475235 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.475859 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.476651 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.476875 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.476984 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:41.479440 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:41.479443 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:41.479445 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:41.479466 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:41.481289 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:41.481869 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:41.481872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:41.481873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:41.481957 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:41.482257 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:41.482260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:41.482261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:41.482431 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:41.482516 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:41.482868 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 20:20:41.483019 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":216,"thread_start_us":118,"threads_started":1}
I20260926 20:20:41.483354 2128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:41.473651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:35875 (local address 127.0.0.1:55800)
0926 20:20:41.473971 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:41.473984 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:41.474085 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:41.474480 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:41.474483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:41.474995 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:41.475110 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:41.475117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.475984 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.475987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:41.476535 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:41.476543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.476987 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.476990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:41.477061 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:41.477687 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:41.477700 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:41.479290 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:41.481389 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:41.481401 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:41.481410 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:41.481754 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:41.482059 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:41.482063 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:41.482067 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:41.482168 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:41.482540 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:41.482548 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:41.482690 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 20:20:41.482824 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 20:20:41.483111 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":89,"thread_start_us":125,"threads_started":1}
I20260926 20:20:41.487711 1581 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:55800:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 20:20:41.488508 1581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:20:41.493697 1719 tablet_service.cc:1467] Processing CreateTablet for tablet 5794f396140648e3bf4c6137206d51ca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=46e9e6ba192444f09291cfa7c416d559]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:20:41.493847 1999 tablet_service.cc:1467] Processing CreateTablet for tablet 5794f396140648e3bf4c6137206d51ca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=46e9e6ba192444f09291cfa7c416d559]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:20:41.494045 1719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5794f396140648e3bf4c6137206d51ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:41.494240 1999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5794f396140648e3bf4c6137206d51ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:41.496433 2089 tablet_bootstrap.cc:492] T 5794f396140648e3bf4c6137206d51ca P bf9282b7099c4965952ce4c04f9c187a: Bootstrap starting.
I20260926 20:20:41.496498 2088 tablet_bootstrap.cc:492] T 5794f396140648e3bf4c6137206d51ca P b416c883c4a24b1f94adc7dfdb110f3a: Bootstrap starting.
I20260926 20:20:41.496830 1859 tablet_service.cc:1467] Processing CreateTablet for tablet 5794f396140648e3bf4c6137206d51ca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=46e9e6ba192444f09291cfa7c416d559]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:20:41.497177 1859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5794f396140648e3bf4c6137206d51ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:41.497262 2089 tablet_bootstrap.cc:654] T 5794f396140648e3bf4c6137206d51ca P bf9282b7099c4965952ce4c04f9c187a: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:41.497429 2088 tablet_bootstrap.cc:654] T 5794f396140648e3bf4c6137206d51ca P b416c883c4a24b1f94adc7dfdb110f3a: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:41.498947 2089 tablet_bootstrap.cc:492] T 5794f396140648e3bf4c6137206d51ca P bf9282b7099c4965952ce4c04f9c187a: No bootstrap required, opened a new log
I20260926 20:20:41.499032 2089 ts_tablet_manager.cc:1397] T 5794f396140648e3bf4c6137206d51ca P bf9282b7099c4965952ce4c04f9c187a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 20:20:41.499462 2089 raft_consensus.cc:348] T 5794f396140648e3bf4c6137206d51ca P bf9282b7099c4965952ce4c04f9c187a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } }
I20260926 20:20:41.499642 2089 raft_consensus.cc:374] T 5794f396140648e3bf4c6137206d51ca P bf9282b7099c4965952ce4c04f9c187a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:41.499681 2089 raft_consensus.cc:729] T 5794f396140648e3bf4c6137206d51ca P bf9282b7099c4965952ce4c04f9c187a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9282b7099c4965952ce4c04f9c187a, State: Initialized, Role: FOLLOWER
I20260926 20:20:41.499747 2090 tablet_bootstrap.cc:492] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4: Bootstrap starting.
I20260926 20:20:41.499811 2089 consensus_queue.cc:260] T 5794f396140648e3bf4c6137206d51ca P bf9282b7099c4965952ce4c04f9c187a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } }
I20260926 20:20:41.500072 2095 tablet_replica.cc:439] T 5794f396140648e3bf4c6137206d51ca P bf9282b7099c4965952ce4c04f9c187a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } } }
I20260926 20:20:41.500213 2095 tablet_replica.cc:442] T 5794f396140648e3bf4c6137206d51ca P bf9282b7099c4965952ce4c04f9c187a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:20:41.500638 2090 tablet_bootstrap.cc:654] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:41.500757 2088 tablet_bootstrap.cc:492] T 5794f396140648e3bf4c6137206d51ca P b416c883c4a24b1f94adc7dfdb110f3a: No bootstrap required, opened a new log
I20260926 20:20:41.500836 2088 ts_tablet_manager.cc:1397] T 5794f396140648e3bf4c6137206d51ca P b416c883c4a24b1f94adc7dfdb110f3a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260926 20:20:41.501339 2088 raft_consensus.cc:348] T 5794f396140648e3bf4c6137206d51ca P b416c883c4a24b1f94adc7dfdb110f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } }
I20260926 20:20:41.501461 2088 raft_consensus.cc:374] T 5794f396140648e3bf4c6137206d51ca P b416c883c4a24b1f94adc7dfdb110f3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:41.501494 2088 raft_consensus.cc:729] T 5794f396140648e3bf4c6137206d51ca P b416c883c4a24b1f94adc7dfdb110f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b416c883c4a24b1f94adc7dfdb110f3a, State: Initialized, Role: FOLLOWER
I20260926 20:20:41.501619 2088 consensus_queue.cc:260] T 5794f396140648e3bf4c6137206d51ca P b416c883c4a24b1f94adc7dfdb110f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } }
I20260926 20:20:41.502102 2094 tablet_replica.cc:439] T 5794f396140648e3bf4c6137206d51ca P b416c883c4a24b1f94adc7dfdb110f3a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } } }
I20260926 20:20:41.502170 2089 ts_tablet_manager.cc:1428] T 5794f396140648e3bf4c6137206d51ca P bf9282b7099c4965952ce4c04f9c187a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 20:20:41.502242 2094 tablet_replica.cc:442] T 5794f396140648e3bf4c6137206d51ca P b416c883c4a24b1f94adc7dfdb110f3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:20:41.502341 2088 ts_tablet_manager.cc:1428] T 5794f396140648e3bf4c6137206d51ca P b416c883c4a24b1f94adc7dfdb110f3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260926 20:20:41.503670 2090 tablet_bootstrap.cc:492] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4: No bootstrap required, opened a new log
I20260926 20:20:41.503763 2090 ts_tablet_manager.cc:1397] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260926 20:20:41.504213 2090 raft_consensus.cc:348] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } }
I20260926 20:20:41.504359 2090 raft_consensus.cc:374] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:41.504400 2090 raft_consensus.cc:729] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4196f4ee33594a9d8d134440c9f000d4, State: Initialized, Role: FOLLOWER
I20260926 20:20:41.504532 2090 consensus_queue.cc:260] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } }
I20260926 20:20:41.504841 2090 ts_tablet_manager.cc:1428] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:20:41.505101 2109 tablet_replica.cc:439] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } } }
I20260926 20:20:41.505256 2109 tablet_replica.cc:442] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:20:41.645751 2109 raft_consensus.cc:482] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:20:41.645907 2109 raft_consensus.cc:504] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } }
I20260926 20:20:41.646401 2109 leader_election.cc:288] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9282b7099c4965952ce4c04f9c187a (127.0.116.193:44459), b416c883c4a24b1f94adc7dfdb110f3a (127.0.116.195:38937)
I20260926 20:20:41.647037 2019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5794f396140648e3bf4c6137206d51ca" candidate_uuid: "4196f4ee33594a9d8d134440c9f000d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" is_pre_election: true
I20260926 20:20:41.647047 1739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5794f396140648e3bf4c6137206d51ca" candidate_uuid: "4196f4ee33594a9d8d134440c9f000d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9282b7099c4965952ce4c04f9c187a" is_pre_election: true
I20260926 20:20:41.647168 2019 raft_consensus.cc:2454] T 5794f396140648e3bf4c6137206d51ca P b416c883c4a24b1f94adc7dfdb110f3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4196f4ee33594a9d8d134440c9f000d4 in term 0.
I20260926 20:20:41.647248 1739 raft_consensus.cc:2454] T 5794f396140648e3bf4c6137206d51ca P bf9282b7099c4965952ce4c04f9c187a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4196f4ee33594a9d8d134440c9f000d4 in term 0.
I20260926 20:20:41.647598 1813 leader_election.cc:302] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4196f4ee33594a9d8d134440c9f000d4, b416c883c4a24b1f94adc7dfdb110f3a; no voters:
I20260926 20:20:41.647797 2109 raft_consensus.cc:2781] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:20:41.647853 2109 raft_consensus.cc:482] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:20:41.647879 2109 raft_consensus.cc:3037] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:41.648890 2109 raft_consensus.cc:504] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } }
I20260926 20:20:41.649247 2109 leader_election.cc:288] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4 [CANDIDATE]: Term 1 election: Requested vote from peers bf9282b7099c4965952ce4c04f9c187a (127.0.116.193:44459), b416c883c4a24b1f94adc7dfdb110f3a (127.0.116.195:38937)
I20260926 20:20:41.649627 2019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5794f396140648e3bf4c6137206d51ca" candidate_uuid: "4196f4ee33594a9d8d134440c9f000d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b416c883c4a24b1f94adc7dfdb110f3a"
I20260926 20:20:41.649619 1739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5794f396140648e3bf4c6137206d51ca" candidate_uuid: "4196f4ee33594a9d8d134440c9f000d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9282b7099c4965952ce4c04f9c187a"
I20260926 20:20:41.649773 2019 raft_consensus.cc:3037] T 5794f396140648e3bf4c6137206d51ca P b416c883c4a24b1f94adc7dfdb110f3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:41.649814 1739 raft_consensus.cc:3037] T 5794f396140648e3bf4c6137206d51ca P bf9282b7099c4965952ce4c04f9c187a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:41.650715 2019 raft_consensus.cc:2454] T 5794f396140648e3bf4c6137206d51ca P b416c883c4a24b1f94adc7dfdb110f3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4196f4ee33594a9d8d134440c9f000d4 in term 1.
I20260926 20:20:41.650732 1739 raft_consensus.cc:2454] T 5794f396140648e3bf4c6137206d51ca P bf9282b7099c4965952ce4c04f9c187a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4196f4ee33594a9d8d134440c9f000d4 in term 1.
I20260926 20:20:41.651028 1813 leader_election.cc:302] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4196f4ee33594a9d8d134440c9f000d4, b416c883c4a24b1f94adc7dfdb110f3a; no voters:
I20260926 20:20:41.651219 2109 raft_consensus.cc:2781] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:41.651437 2109 raft_consensus.cc:686] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4 [term 1 LEADER]: Becoming Leader. State: Replica: 4196f4ee33594a9d8d134440c9f000d4, State: Running, Role: LEADER
I20260926 20:20:41.651626 2109 consensus_queue.cc:237] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } }
I20260926 20:20:41.652336 2133 tablet_replica.cc:439] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4: TxnStatusTablet state changed. Reason: New leader 4196f4ee33594a9d8d134440c9f000d4. Latest consensus state: current_term: 1 leader_uuid: "4196f4ee33594a9d8d134440c9f000d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } } }
I20260926 20:20:41.652487 2133 tablet_replica.cc:442] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:20:41.652704 2135 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 20:20:41.653071 2135 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 20:20:41.653445 1582 catalog_manager.cc:5495] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4196f4ee33594a9d8d134440c9f000d4 (127.0.116.194). New cstate: current_term: 1 leader_uuid: "4196f4ee33594a9d8d134440c9f000d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } health_report { overall_health: UNKNOWN } } }
I20260926 20:20:41.695648 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:41.692189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47886 (local address 127.0.116.194:45943)
0926 20:20:41.692313 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:41.692329 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:41.692408 (+ 79us) server_negotiation.cc:407] Connection header received
0926 20:20:41.692571 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:41.692574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:41.692616 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:41.692682 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:20:41.692957 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.693447 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.694613 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.694819 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.694897 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:41.695210 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:20:41.695273 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:20:41.695331 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:41.695443 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 20:20:41.695484 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:20:41.695672 2128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:41.692090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:45943 (local address 127.0.0.1:47886)
0926 20:20:41.692334 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:41.692349 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:41.692462 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:41.692729 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:41.692732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:41.692740 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:20:41.692850 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:41.692855 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.693541 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.693545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:41.694507 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:41.694517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.694920 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.694923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:41.695017 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:41.695070 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:20:41.695347 (+ 277us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:20:41.695360 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 20:20:41.695428 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 20:20:41.695531 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260926 20:20:41.698256 2019 raft_consensus.cc:1261] T 5794f396140648e3bf4c6137206d51ca P b416c883c4a24b1f94adc7dfdb110f3a [term 1 FOLLOWER]: Refusing update from remote peer 4196f4ee33594a9d8d134440c9f000d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:41.698282 1739 raft_consensus.cc:1261] T 5794f396140648e3bf4c6137206d51ca P bf9282b7099c4965952ce4c04f9c187a [term 1 FOLLOWER]: Refusing update from remote peer 4196f4ee33594a9d8d134440c9f000d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:41.698683 2133 consensus_queue.cc:1035] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:20:41.698835 2109 consensus_queue.cc:1035] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:20:41.700145 2095 tablet_replica.cc:439] T 5794f396140648e3bf4c6137206d51ca P bf9282b7099c4965952ce4c04f9c187a: TxnStatusTablet state changed. Reason: New leader 4196f4ee33594a9d8d134440c9f000d4. Latest consensus state: current_term: 1 leader_uuid: "4196f4ee33594a9d8d134440c9f000d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } } }
I20260926 20:20:41.700251 2095 tablet_replica.cc:442] T 5794f396140648e3bf4c6137206d51ca P bf9282b7099c4965952ce4c04f9c187a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:20:41.700199 2094 tablet_replica.cc:439] T 5794f396140648e3bf4c6137206d51ca P b416c883c4a24b1f94adc7dfdb110f3a: TxnStatusTablet state changed. Reason: New leader 4196f4ee33594a9d8d134440c9f000d4. Latest consensus state: current_term: 1 leader_uuid: "4196f4ee33594a9d8d134440c9f000d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } } }
I20260926 20:20:41.700299 2094 tablet_replica.cc:442] T 5794f396140648e3bf4c6137206d51ca P b416c883c4a24b1f94adc7dfdb110f3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:20:41.702507 2109 tablet_replica.cc:439] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4196f4ee33594a9d8d134440c9f000d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } } }
I20260926 20:20:41.702610 2109 tablet_replica.cc:442] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:20:41.702687 2138 mvcc.cc:205] Tried to move back new op lower bound from 7333699754791600128 to 7333699754606534656. Current Snapshot: MvccSnapshot[applied={T|T < 7333699754791600128}]
I20260926 20:20:41.702896 2095 tablet_replica.cc:439] T 5794f396140648e3bf4c6137206d51ca P bf9282b7099c4965952ce4c04f9c187a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4196f4ee33594a9d8d134440c9f000d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } } }
I20260926 20:20:41.702997 2095 tablet_replica.cc:442] T 5794f396140648e3bf4c6137206d51ca P bf9282b7099c4965952ce4c04f9c187a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:20:41.703764 2139 mvcc.cc:205] Tried to move back new op lower bound from 7333699754791600128 to 7333699754606534656. Current Snapshot: MvccSnapshot[applied={T|T < 7333699754791600128}]
I20260926 20:20:41.703814 2133 tablet_replica.cc:439] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4196f4ee33594a9d8d134440c9f000d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } } }
I20260926 20:20:41.703905 2133 tablet_replica.cc:442] T 5794f396140648e3bf4c6137206d51ca P 4196f4ee33594a9d8d134440c9f000d4: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:20:41.705083 2094 tablet_replica.cc:439] T 5794f396140648e3bf4c6137206d51ca P b416c883c4a24b1f94adc7dfdb110f3a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4196f4ee33594a9d8d134440c9f000d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 } } peers { permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 } } peers { permanent_uuid: "b416c883c4a24b1f94adc7dfdb110f3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38937 } } }
I20260926 20:20:41.705188 2094 tablet_replica.cc:442] T 5794f396140648e3bf4c6137206d51ca P b416c883c4a24b1f94adc7dfdb110f3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:20:41.713195 2149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:41.709351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48012 (local address 127.0.116.195:38937)
0926 20:20:41.709553 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:41.709558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:41.709637 (+ 79us) server_negotiation.cc:407] Connection header received
0926 20:20:41.709829 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:41.709832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:41.709896 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:41.710028 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:20:41.710254 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.710780 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.711697 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.711954 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.712069 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:41.712336 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:20:41.712794 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:20:41.712936 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:41.712985 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 20:20:41.713038 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":68,"threads_started":1}
I20260926 20:20:41.724087 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:41.720448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:45943 (local address 127.0.116.195:38969)
0926 20:20:41.720724 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:41.720737 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:41.720847 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:41.721098 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:41.721100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:41.721106 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:20:41.721202 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:41.721206 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.722179 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.722181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:41.722790 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:41.722798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.723337 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.723340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:41.723403 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:41.723445 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:20:41.723814 (+ 369us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:20:41.723837 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:20:41.723886 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 20:20:41.723951 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":107,"threads_started":1}
I20260926 20:20:41.724082 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:41.720501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:38969 (local address 127.0.116.194:45943)
0926 20:20:41.720631 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:41.720636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:41.720809 (+ 173us) server_negotiation.cc:407] Connection header received
0926 20:20:41.720946 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:41.720949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:41.721004 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:41.721084 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:20:41.721300 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.722078 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.722914 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.723210 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.723314 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:41.723620 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:20:41.723705 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:20:41.723807 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:41.723900 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 20:20:41.723953 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:20:41.731119 1739 raft_consensus.cc:1261] T 494bdf63e1114301aee16972a5c6f04b P bf9282b7099c4965952ce4c04f9c187a [term 1 FOLLOWER]: Refusing update from remote peer b416c883c4a24b1f94adc7dfdb110f3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:41.731269 1879 raft_consensus.cc:1261] T 494bdf63e1114301aee16972a5c6f04b P 4196f4ee33594a9d8d134440c9f000d4 [term 1 FOLLOWER]: Refusing update from remote peer b416c883c4a24b1f94adc7dfdb110f3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:41.731657 2121 consensus_queue.cc:1035] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9282b7099c4965952ce4c04f9c187a" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:20:41.731844 2094 consensus_queue.cc:1035] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4196f4ee33594a9d8d134440c9f000d4" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:20:41.755656 2149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:41.751733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:44679 (local address 127.0.116.195:38937)
0926 20:20:41.751875 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:41.751879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:41.752130 (+ 251us) server_negotiation.cc:407] Connection header received
0926 20:20:41.752218 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:41.752221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:41.752262 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:41.752331 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:20:41.752572 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.753286 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.754463 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.754736 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.754854 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:41.755141 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:20:41.755228 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:20:41.755343 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:41.755433 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 20:20:41.755511 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:20:41.755656 2156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:41.751651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:38937 (local address 127.0.116.194:44679)
0926 20:20:41.751999 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:41.752015 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:41.752127 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:41.752364 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:41.752367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:41.752376 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:20:41.752484 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:41.752491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.753401 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.753407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:41.754345 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:41.754353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.754855 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.754858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:41.754945 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:41.755004 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:20:41.755348 (+ 344us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:20:41.755357 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 20:20:41.755425 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 20:20:41.755533 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":129,"threads_started":1}
I20260926 20:20:41.785346 1582 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:20:41.785534 1582 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:20:41.788013 1582 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 009abcca0dcb4e6db8081ef8c2af03c3: Sending DeleteTablet for 3 replicas of tablet 494bdf63e1114301aee16972a5c6f04b
I20260926 20:20:41.788820 1719 tablet_service.cc:1514] Processing DeleteTablet for tablet 494bdf63e1114301aee16972a5c6f04b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:41 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:55318
I20260926 20:20:41.788895 1999 tablet_service.cc:1514] Processing DeleteTablet for tablet 494bdf63e1114301aee16972a5c6f04b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:41 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:43226
I20260926 20:20:41.789343 1859 tablet_service.cc:1514] Processing DeleteTablet for tablet 494bdf63e1114301aee16972a5c6f04b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:41 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:32774
I20260926 20:20:41.789739 2160 tablet_replica.cc:330] T 494bdf63e1114301aee16972a5c6f04b P bf9282b7099c4965952ce4c04f9c187a: stopping tablet replica
I20260926 20:20:41.789894 2160 raft_consensus.cc:2229] T 494bdf63e1114301aee16972a5c6f04b P bf9282b7099c4965952ce4c04f9c187a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:20:41.790057 2160 raft_consensus.cc:2258] T 494bdf63e1114301aee16972a5c6f04b P bf9282b7099c4965952ce4c04f9c187a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:20:41.790365 2162 tablet_replica.cc:330] T 494bdf63e1114301aee16972a5c6f04b P 4196f4ee33594a9d8d134440c9f000d4: stopping tablet replica
I20260926 20:20:41.790508 2162 raft_consensus.cc:2229] T 494bdf63e1114301aee16972a5c6f04b P 4196f4ee33594a9d8d134440c9f000d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:20:41.790632 2162 raft_consensus.cc:2258] T 494bdf63e1114301aee16972a5c6f04b P 4196f4ee33594a9d8d134440c9f000d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:20:41.790934 2160 ts_tablet_manager.cc:1905] T 494bdf63e1114301aee16972a5c6f04b P bf9282b7099c4965952ce4c04f9c187a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:20:41.792024 2162 ts_tablet_manager.cc:1905] T 494bdf63e1114301aee16972a5c6f04b P 4196f4ee33594a9d8d134440c9f000d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:20:41.794533 2160 ts_tablet_manager.cc:1918] T 494bdf63e1114301aee16972a5c6f04b P bf9282b7099c4965952ce4c04f9c187a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:20:41.794682 2160 log.cc:1196] T 494bdf63e1114301aee16972a5c6f04b P bf9282b7099c4965952ce4c04f9c187a: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-0/wal/wals/494bdf63e1114301aee16972a5c6f04b
I20260926 20:20:41.794817 2162 ts_tablet_manager.cc:1918] T 494bdf63e1114301aee16972a5c6f04b P 4196f4ee33594a9d8d134440c9f000d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:20:41.794901 2162 log.cc:1196] T 494bdf63e1114301aee16972a5c6f04b P 4196f4ee33594a9d8d134440c9f000d4: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/494bdf63e1114301aee16972a5c6f04b
I20260926 20:20:41.795143 2160 ts_tablet_manager.cc:1939] T 494bdf63e1114301aee16972a5c6f04b P bf9282b7099c4965952ce4c04f9c187a: Deleting consensus metadata
I20260926 20:20:41.795171 2162 ts_tablet_manager.cc:1939] T 494bdf63e1114301aee16972a5c6f04b P 4196f4ee33594a9d8d134440c9f000d4: Deleting consensus metadata
I20260926 20:20:41.795899 1568 catalog_manager.cc:4841] TS 4196f4ee33594a9d8d134440c9f000d4 (127.0.116.194:45943): tablet 494bdf63e1114301aee16972a5c6f04b (table test-table [id=ce858237edc240b1b547a52e04bd552a]) successfully deleted
I20260926 20:20:41.796021 1567 catalog_manager.cc:4841] TS bf9282b7099c4965952ce4c04f9c187a (127.0.116.193:44459): tablet 494bdf63e1114301aee16972a5c6f04b (table test-table [id=ce858237edc240b1b547a52e04bd552a]) successfully deleted
I20260926 20:20:41.796298 2161 tablet_replica.cc:330] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a: stopping tablet replica
I20260926 20:20:41.796438 2161 raft_consensus.cc:2229] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:20:41.796744 2161 raft_consensus.cc:2258] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:20:41.797425 2161 ts_tablet_manager.cc:1905] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:20:41.800123 2161 ts_tablet_manager.cc:1918] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:20:41.800199 2161 log.cc:1196] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/494bdf63e1114301aee16972a5c6f04b
I20260926 20:20:41.800467 2161 ts_tablet_manager.cc:1939] T 494bdf63e1114301aee16972a5c6f04b P b416c883c4a24b1f94adc7dfdb110f3a: Deleting consensus metadata
I20260926 20:20:41.808094 1569 catalog_manager.cc:4841] TS b416c883c4a24b1f94adc7dfdb110f3a (127.0.116.195:38937): tablet 494bdf63e1114301aee16972a5c6f04b (table test-table [id=ce858237edc240b1b547a52e04bd552a]) successfully deleted
Sep 26 20:20:41 dist-test-slave-qsm9 krb5kdc[1537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454040, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.116.193@KRBTEST.COM
I20260926 20:20:41.813963 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:41.791077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34132 (local address 127.0.116.193:44459)
0926 20:20:41.791186 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:41.791190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:41.800956 (+ 9766us) server_negotiation.cc:407] Connection header received
0926 20:20:41.801172 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:41.801175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:41.801228 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:41.801280 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:41.802593 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.803410 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.804693 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.805028 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.805098 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:41.809805 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:41.809821 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:41.809840 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:41.809870 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:41.811884 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:41.812385 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:41.812392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:41.812394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:41.812492 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:41.812795 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:41.812799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:41.812801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:41.813105 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:41.813283 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:41.813652 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 20:20:41.813777 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260926 20:20:41.814795 1779 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34132
I20260926 20:20:41.826821 2129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:41.817123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55806 (local address 127.0.116.254:35875)
0926 20:20:41.817243 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:41.817247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:41.817699 (+ 452us) server_negotiation.cc:407] Connection header received
0926 20:20:41.817876 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:41.817880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:41.817930 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:41.817982 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:41.819245 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.820197 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.821362 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:41.821717 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:41.821790 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:41.823534 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:41.823538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:41.823541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:41.823567 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:41.825126 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:41.825603 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:41.825606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:41.825609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:41.825691 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:41.825996 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:41.825998 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:41.825999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:41.826187 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:41.826300 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:41.826555 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 20:20:41.826661 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 20:20:41.827288 1582 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55806
I20260926 20:20:41.828845 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 1654
I20260926 20:20:41.835831 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 1794
I20260926 20:20:41.842214 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 1934
I20260926 20:20:41.848260 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 1550
2026-09-26T20:20:41Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3221 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:41 dist-test-slave-qsm9 krb5kdc[2169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:41 dist-test-slave-qsm9 krb5kdc[2169](info): set up 2 sockets
Sep 26 20:20:41 dist-test-slave-qsm9 krb5kdc[2169](info): commencing operation
krb5kdc: starting...
W20260926 20:20:42.895601 467 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:20:42 dist-test-slave-qsm9 krb5kdc[2169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:20:43.023101 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:44587
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:35035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:44587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:43.137388 2182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:43.137737 2182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:43.137825 2182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:43.141963 2182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:43.142062 2182 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 20:20:43.142092 2182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:43.142138 2182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:43.142175 2182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:43.142204 2182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:43.147653 2182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:35035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:44587
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:44587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:43.148838 2182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:43.149938 2182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:43.158500 2188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:43.158572 2182 server_base.cc:1034] running on GCE node
W20260926 20:20:43.158558 2190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:43.158514 2187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:43.159301 2182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:43.160432 2182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:43.161597 2182 hybrid_clock.cc:648] HybridClock initialized: now 1790454043161577 us; error 30 us; skew 500 ppm
Sep 26 20:20:43 dist-test-slave-qsm9 krb5kdc[2169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:43.164316 2182 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:20:43.165182 2182 webserver.cc:460] Webserver started at http://127.0.116.254:37631/ using document root <none> and password file <none>
I20260926 20:20:43.165686 2182 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:43.165746 2182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:43.165906 2182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:43.167946 2182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "e4137e3e36eb477b87fee13fafe0d972"
format_stamp: "Formatted at 2026-09-26 20:20:43 on dist-test-slave-qsm9"
server_key: "df12a4da9e2936d443444833e97f0a4d"
server_key_iv: "6ce789d85c067fe93a40cc9ac717a23f"
server_key_version: "encryptionkey@0"
I20260926 20:20:43.168445 2182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "e4137e3e36eb477b87fee13fafe0d972"
format_stamp: "Formatted at 2026-09-26 20:20:43 on dist-test-slave-qsm9"
server_key: "df12a4da9e2936d443444833e97f0a4d"
server_key_iv: "6ce789d85c067fe93a40cc9ac717a23f"
server_key_version: "encryptionkey@0"
I20260926 20:20:43.172003 2182 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:20:43.174366 2197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:43.175786 2182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:20:43.175921 2182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "e4137e3e36eb477b87fee13fafe0d972"
format_stamp: "Formatted at 2026-09-26 20:20:43 on dist-test-slave-qsm9"
server_key: "df12a4da9e2936d443444833e97f0a4d"
server_key_iv: "6ce789d85c067fe93a40cc9ac717a23f"
server_key_version: "encryptionkey@0"
I20260926 20:20:43.176038 2182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:43.186733 2182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:43.189522 2182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:43.189728 2182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:43.197695 2182 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:44587
I20260926 20:20:43.197700 2259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:44587 every 8 connection(s)
I20260926 20:20:43.198652 2182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:43.200035 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 2182
I20260926 20:20:43.200150 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:20:43.200460 467 external_mini_cluster.cc:1428] Setting key f5388ef0b4031cfe696e6219c3552067
I20260926 20:20:43.201895 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:43.207494 2260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972: Bootstrap starting.
Sep 26 20:20:43 dist-test-slave-qsm9 krb5kdc[2169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:43.210026 2260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:43.211306 2260 log.cc:824] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:43.213302 2260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972: No bootstrap required, opened a new log
I20260926 20:20:43.215186 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:43.201920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.0.116.254:44587)
0926 20:20:43.202364 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:43.202380 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:43.202417 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:20:43.203084 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:43.203104 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:43.203444 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:43.203860 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:43.204684 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.205684 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.206564 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.207041 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.207211 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:43.209979 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:43.209985 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:43.209998 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:43.210026 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:43.212239 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:43.212707 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:43.212714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:43.212720 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:43.212859 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:43.213150 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:43.213154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:43.213156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:43.213637 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:43.213854 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:43.214177 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 20:20:43.214478 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":136,"threads_started":1}
I20260926 20:20:43.216568 2260 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4137e3e36eb477b87fee13fafe0d972" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44587 } }
I20260926 20:20:43.216784 2260 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:43.216849 2260 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4137e3e36eb477b87fee13fafe0d972, State: Initialized, Role: FOLLOWER
I20260926 20:20:43.217317 2260 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4137e3e36eb477b87fee13fafe0d972" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44587 } }
I20260926 20:20:43.217473 2260 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:43.217551 2260 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:43.217629 2260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:43.218482 2260 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4137e3e36eb477b87fee13fafe0d972" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44587 } }
I20260926 20:20:43.218768 2260 leader_election.cc:302] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4137e3e36eb477b87fee13fafe0d972; no voters:
I20260926 20:20:43.219060 2260 leader_election.cc:288] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:43.219141 2265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:43.219318 2265 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972 [term 1 LEADER]: Becoming Leader. State: Replica: e4137e3e36eb477b87fee13fafe0d972, State: Running, Role: LEADER
I20260926 20:20:43.219645 2265 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4137e3e36eb477b87fee13fafe0d972" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44587 } }
I20260926 20:20:43.220389 2260 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:43.221446 2266 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4137e3e36eb477b87fee13fafe0d972" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4137e3e36eb477b87fee13fafe0d972" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44587 } } }
I20260926 20:20:43.221563 2266 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:43.221920 2267 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4137e3e36eb477b87fee13fafe0d972. Latest consensus state: current_term: 1 leader_uuid: "e4137e3e36eb477b87fee13fafe0d972" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4137e3e36eb477b87fee13fafe0d972" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44587 } } }
I20260926 20:20:43.222527 2267 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972 [sys.catalog]: This master's current role is: LEADER
W20260926 20:20:43.225508 2281 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:20:43.225577 2281 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:20:43.225678 2274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:43.226420 2274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:43.231842 2274 catalog_manager.cc:1347] Generated new cluster ID: 90c21e45ee6c4b9bad2cda366908c937
I20260926 20:20:43.231910 2274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:43.239724 2274 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:43.240950 2274 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:43.255585 2274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972: Generated new TSK 0
I20260926 20:20:43.256500 2274 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
I20260926 20:20:43.286679 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44587
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:35035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:43.403537 2286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:43.403815 2286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:43.403908 2286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:43.408124 2286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:43.408182 2286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:43.408252 2286 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 20:20:43.408281 2286 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:20:43.408334 2286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:20:43.413771 2286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:35035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.116.254:44587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:43.414912 2286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:43.416047 2286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:43.425875 2294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:43.426061 2286 server_base.cc:1034] running on GCE node
W20260926 20:20:43.425875 2292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:43.425895 2291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:43.426759 2286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:43.427438 2286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:43.428651 2286 hybrid_clock.cc:648] HybridClock initialized: now 1790454043428623 us; error 37 us; skew 500 ppm
Sep 26 20:20:43 dist-test-slave-qsm9 krb5kdc[2169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:43.431789 2286 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:20:43.432757 2286 webserver.cc:460] Webserver started at http://127.0.116.193:36363/ using document root <none> and password file <none>
I20260926 20:20:43.433303 2286 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:43.433363 2286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:43.433542 2286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:43.435757 2286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "0dbe95cf940549da948f8adf2256841b"
format_stamp: "Formatted at 2026-09-26 20:20:43 on dist-test-slave-qsm9"
server_key: "8be78e4943b1042e37d24c5f99cfcfb4"
server_key_iv: "456a9ae674eed75043c7adb67958ec4c"
server_key_version: "encryptionkey@0"
I20260926 20:20:43.436327 2286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "0dbe95cf940549da948f8adf2256841b"
format_stamp: "Formatted at 2026-09-26 20:20:43 on dist-test-slave-qsm9"
server_key: "8be78e4943b1042e37d24c5f99cfcfb4"
server_key_iv: "456a9ae674eed75043c7adb67958ec4c"
server_key_version: "encryptionkey@0"
I20260926 20:20:43.440408 2286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:20:43.443275 2301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:43.444653 2286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:20:43.444802 2286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "0dbe95cf940549da948f8adf2256841b"
format_stamp: "Formatted at 2026-09-26 20:20:43 on dist-test-slave-qsm9"
server_key: "8be78e4943b1042e37d24c5f99cfcfb4"
server_key_iv: "456a9ae674eed75043c7adb67958ec4c"
server_key_version: "encryptionkey@0"
I20260926 20:20:43.444959 2286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:43.486366 2286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:43.489902 2286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:43.490116 2286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:43.491636 2286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:43.491703 2286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:43.491773 2286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:43.491833 2286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:20:43 dist-test-slave-qsm9 krb5kdc[2169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:43.503119 2286 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:35897
I20260926 20:20:43.503984 2416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:35897 every 8 connection(s)
I20260926 20:20:43.504113 2286 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:20:43.506279 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:43.492557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:50013 (local address 127.0.116.254:44587)
0926 20:20:43.492688 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:43.492692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:43.493817 (+ 1125us) server_negotiation.cc:407] Connection header received
0926 20:20:43.494998 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:43.495001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:43.495063 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:43.495152 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:43.496625 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.497189 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.498569 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.498996 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.499130 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:43.502060 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:43.502064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:43.502067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:43.502096 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:43.503778 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:43.504386 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:43.504389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:43.504392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:43.504479 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:43.505140 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:43.505143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:43.505145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:43.505383 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:43.505489 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:43.505975 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 20:20:43.506107 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:20:43.507052 2310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:43.493139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44587 (local address 127.0.116.193:50013)
0926 20:20:43.493663 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:43.493707 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:43.494715 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:43.495353 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:43.495360 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:43.495837 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:43.496473 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:43.496491 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.497349 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.497352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:43.498434 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:43.498443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.499182 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.499188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:43.499430 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:43.500092 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:43.500108 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:43.501904 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:43.503897 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:43.503903 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:43.503918 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:43.504266 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:43.504714 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:43.504716 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:43.504718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:43.504849 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:43.505510 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:43.505516 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:43.505742 (+ 226us) client_negotiation.cc:769] Sending connection context
0926 20:20:43.505935 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 20:20:43.506247 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":147,"threads_started":1}
I20260926 20:20:43.508502 2417 heartbeater.cc:347] Connected to a master server at 127.0.116.254:44587
I20260926 20:20:43.508667 2417 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:43.509248 2417 heartbeater.cc:511] Master 127.0.116.254:44587 requested a full tablet report, sending...
I20260926 20:20:43.510898 2213 ts_manager.cc:194] Registered new tserver with Master: 0dbe95cf940549da948f8adf2256841b (127.0.116.193:35897)
I20260926 20:20:43.512292 2213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:50013
I20260926 20:20:43.513959 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 2286
I20260926 20:20:43.514092 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:20:43.514469 467 external_mini_cluster.cc:1428] Setting key a1cda463699b2e041df86675b3e5e59e
I20260926 20:20:43.518512 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:43.510742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:37265 (local address 127.0.116.254:44587)
0926 20:20:43.510881 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:43.510885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:43.511053 (+ 168us) server_negotiation.cc:407] Connection header received
0926 20:20:43.511235 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:43.511238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:43.511280 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:43.511349 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:43.511881 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.512431 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.513344 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.513566 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.513737 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:43.515100 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:43.515106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:43.515109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:43.515136 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:43.516816 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:43.517251 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:43.517257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:43.517262 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:43.517354 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:43.517606 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:43.517612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:43.517616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:43.517869 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:43.517980 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:43.518294 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 20:20:43.518397 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:20:43.518664 2423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:43.510678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44587 (local address 127.0.116.193:37265)
0926 20:20:43.510970 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:43.510988 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:43.511118 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:43.511391 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:43.511394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:43.511640 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:43.511761 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:43.511769 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.512573 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.512576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:43.513190 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:43.513199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.513689 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.513693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:43.513844 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:43.514442 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:43.514456 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:43.514922 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:43.516958 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:43.516960 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:43.516962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:43.517156 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:43.517443 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:43.517445 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:43.517447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:43.517522 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:43.517980 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:43.517998 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:43.518114 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 20:20:43.518241 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 20:20:43.518404 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.0.116.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
I20260926 20:20:43.545055 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44587
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:35035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:43.656395 2426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:43.656679 2426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:43.656728 2426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:43.660872 2426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:43.660940 2426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:43.660981 2426 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 20:20:43.661024 2426 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:20:43.661092 2426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:20:43.666446 2426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:35035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.116.254:44587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:43.667537 2426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:43.668560 2426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:43.678123 2432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:43.678121 2434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:43.678121 2431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:43.678418 2426 server_base.cc:1034] running on GCE node
I20260926 20:20:43.678952 2426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:43.679688 2426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:43.680891 2426 hybrid_clock.cc:648] HybridClock initialized: now 1790454043680880 us; error 33 us; skew 500 ppm
Sep 26 20:20:43 dist-test-slave-qsm9 krb5kdc[2169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:43.683473 2426 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:20:43.684376 2426 webserver.cc:460] Webserver started at http://127.0.116.194:38645/ using document root <none> and password file <none>
I20260926 20:20:43.684939 2426 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:43.685032 2426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:43.685262 2426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:43.687314 2426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "d99c92f36fb3458582f7aaab81182991"
format_stamp: "Formatted at 2026-09-26 20:20:43 on dist-test-slave-qsm9"
server_key: "ff822f77d911451cae2e1287dbcdf5f7"
server_key_iv: "e42772bd651d295f4b00ee4978e4744a"
server_key_version: "encryptionkey@0"
I20260926 20:20:43.687930 2426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "d99c92f36fb3458582f7aaab81182991"
format_stamp: "Formatted at 2026-09-26 20:20:43 on dist-test-slave-qsm9"
server_key: "ff822f77d911451cae2e1287dbcdf5f7"
server_key_iv: "e42772bd651d295f4b00ee4978e4744a"
server_key_version: "encryptionkey@0"
I20260926 20:20:43.691452 2426 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:20:43.693909 2441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:43.695096 2426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:20:43.695238 2426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "d99c92f36fb3458582f7aaab81182991"
format_stamp: "Formatted at 2026-09-26 20:20:43 on dist-test-slave-qsm9"
server_key: "ff822f77d911451cae2e1287dbcdf5f7"
server_key_iv: "e42772bd651d295f4b00ee4978e4744a"
server_key_version: "encryptionkey@0"
I20260926 20:20:43.695353 2426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:43.722023 2426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:43.725703 2426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:43.725927 2426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:43.727429 2426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:43.727540 2426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:43.727619 2426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:43.727672 2426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:43.738384 2426 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:38091
I20260926 20:20:43.738418 2556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:38091 every 8 connection(s)
Sep 26 20:20:43 dist-test-slave-qsm9 krb5kdc[2169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:43.739538 2426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:20:43.741662 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 2426
I20260926 20:20:43.741775 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:20:43.742116 467 external_mini_cluster.cc:1428] Setting key d5a8055df33b6f36840438adf1e7dfdd
I20260926 20:20:43.743120 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:43.728333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:32953 (local address 127.0.116.254:44587)
0926 20:20:43.728481 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:43.728486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:43.729499 (+ 1013us) server_negotiation.cc:407] Connection header received
0926 20:20:43.730668 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:43.730675 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:43.730749 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:43.730868 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:43.732742 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.733641 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.735574 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.735804 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.735917 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:43.739219 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:43.739222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:43.739225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:43.739254 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:43.740956 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:43.741453 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:43.741457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:43.741461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:43.741566 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:43.741977 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:43.741981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:43.741983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:43.742228 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:43.742339 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:43.742842 (+ 503us) server_negotiation.cc:299] Negotiation successful
0926 20:20:43.742975 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:20:43.744074 2450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:43.728782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44587 (local address 127.0.116.194:32953)
0926 20:20:43.729321 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:43.729382 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:43.730410 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:43.731207 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:43.731217 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:43.731780 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:43.732546 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:43.732568 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.733808 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.733813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:43.735392 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:43.735403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.735997 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.736003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:43.736245 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:43.737015 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:43.737040 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:43.738990 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:43.741059 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:43.741064 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:43.741075 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:43.741334 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:43.741662 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:43.741664 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:43.741666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:43.741877 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:43.742331 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:43.742337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:43.742565 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 20:20:43.742807 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 20:20:43.743097 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":133,"threads_started":1}
I20260926 20:20:43.745529 2557 heartbeater.cc:347] Connected to a master server at 127.0.116.254:44587
I20260926 20:20:43.745713 2557 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:43.746325 2557 heartbeater.cc:511] Master 127.0.116.254:44587 requested a full tablet report, sending...
I20260926 20:20:43.747581 2214 ts_manager.cc:194] Registered new tserver with Master: d99c92f36fb3458582f7aaab81182991 (127.0.116.194:38091)
I20260926 20:20:43.748487 2214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:32953
WARNING: no policy specified for kudu/127.0.116.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.195@KRBTEST.COM" created.
I20260926 20:20:43.759785 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:43.751997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:40271 (local address 127.0.116.254:44587)
0926 20:20:43.752119 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:43.752125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:43.752363 (+ 238us) server_negotiation.cc:407] Connection header received
0926 20:20:43.752446 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:43.752449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:43.752492 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:43.752559 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:43.752976 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.753483 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.754314 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.754590 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.754675 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:43.756200 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:43.756206 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:43.756213 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:43.756248 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:43.757804 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:43.758355 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:43.758358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:43.758359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:43.758435 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:43.758744 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:43.758746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:43.758747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:43.759070 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:43.759152 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:43.759485 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 20:20:43.759642 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:20:43.760039 2564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:43.751905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44587 (local address 127.0.116.194:40271)
0926 20:20:43.752271 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:43.752283 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:43.752372 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:43.752586 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:43.752588 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:43.752753 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:43.752860 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:43.752865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.753607 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.753610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:43.754168 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:43.754176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.754708 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.754714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:43.754828 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:43.755519 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:43.755536 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:43.756035 (+ 499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:43.757930 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:43.757937 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:43.757942 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:43.758231 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:43.758527 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:43.758533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:43.758538 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:43.758647 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:43.759180 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:43.759185 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:43.759303 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 20:20:43.759435 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 20:20:43.759656 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
I20260926 20:20:43.772672 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44587
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:35035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:43.884368 2566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:43.884671 2566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:43.884748 2566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:43.890533 2566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:43.890586 2566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:43.890657 2566 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 20:20:43.890687 2566 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:20:43.890749 2566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:20:43.896245 2566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:35035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.116.254:44587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:43.897357 2566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:43.898490 2566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:43.908650 2571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:43.908636 2572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:43.908697 2566 server_base.cc:1034] running on GCE node
W20260926 20:20:43.908636 2574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:43.909382 2566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:43.910081 2566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:43.911295 2566 hybrid_clock.cc:648] HybridClock initialized: now 1790454043911278 us; error 32 us; skew 500 ppm
Sep 26 20:20:43 dist-test-slave-qsm9 krb5kdc[2169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:43.914505 2566 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:20:43.915561 2566 webserver.cc:460] Webserver started at http://127.0.116.195:43703/ using document root <none> and password file <none>
I20260926 20:20:43.916136 2566 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:43.916224 2566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:43.916461 2566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:43.918470 2566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "bd1532df667044eeb45599ac193e9617"
format_stamp: "Formatted at 2026-09-26 20:20:43 on dist-test-slave-qsm9"
server_key: "f56eaa03fb5b33c0bdff82c615e8bbc1"
server_key_iv: "52e1ec4c64fcbed8a133be976b3ead88"
server_key_version: "encryptionkey@0"
I20260926 20:20:43.919050 2566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "bd1532df667044eeb45599ac193e9617"
format_stamp: "Formatted at 2026-09-26 20:20:43 on dist-test-slave-qsm9"
server_key: "f56eaa03fb5b33c0bdff82c615e8bbc1"
server_key_iv: "52e1ec4c64fcbed8a133be976b3ead88"
server_key_version: "encryptionkey@0"
I20260926 20:20:43.923050 2566 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:20:43.925572 2581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:43.926803 2566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:20:43.926940 2566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "bd1532df667044eeb45599ac193e9617"
format_stamp: "Formatted at 2026-09-26 20:20:43 on dist-test-slave-qsm9"
server_key: "f56eaa03fb5b33c0bdff82c615e8bbc1"
server_key_iv: "52e1ec4c64fcbed8a133be976b3ead88"
server_key_version: "encryptionkey@0"
I20260926 20:20:43.927057 2566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:43.969420 2566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:43.972551 2566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:43.972776 2566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:43.974308 2566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:43.974390 2566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:43.974467 2566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:43.974509 2566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:20:43 dist-test-slave-qsm9 krb5kdc[2169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:43.985710 2566 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:41685
I20260926 20:20:43.985777 2696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:41685 every 8 connection(s)
I20260926 20:20:43.986891 2566 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:20:43.989374 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:43.975349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:60695 (local address 127.0.116.254:44587)
0926 20:20:43.975484 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:43.975487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:43.976621 (+ 1134us) server_negotiation.cc:407] Connection header received
0926 20:20:43.977698 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:43.977701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:43.977771 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:43.977872 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:43.979603 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.980319 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.981708 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.982045 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.982170 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:43.985566 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:43.985570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:43.985572 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:43.985600 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:43.987180 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:43.987821 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:43.987824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:43.987825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:43.987907 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:43.988256 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:43.988259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:43.988261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:43.988461 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:43.988569 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:43.989073 (+ 504us) server_negotiation.cc:299] Negotiation successful
0926 20:20:43.989200 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:20:43.989856 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 2566
I20260926 20:20:43.989982 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:20:43.990255 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:43.975979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44587 (local address 127.0.116.195:60695)
0926 20:20:43.976469 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:43.976513 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:43.977453 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:43.978056 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:43.978066 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:43.978612 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:43.979364 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:43.979387 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.980470 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.980473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:43.981591 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:43.981604 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.982223 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.982226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:43.982425 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:43.983682 (+ 1257us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:43.983707 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:43.985313 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:43.987306 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:43.987313 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:43.987330 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:43.987689 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:43.988002 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:43.988007 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:43.988009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:43.988153 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:43.988567 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:43.988574 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:43.988836 (+ 262us) client_negotiation.cc:769] Sending connection context
0926 20:20:43.989020 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 20:20:43.989348 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":139,"threads_started":1}
I20260926 20:20:43.990315 467 external_mini_cluster.cc:1428] Setting key df448029d17119ea97d5a8ec3fc291eb
I20260926 20:20:43.991819 2697 heartbeater.cc:347] Connected to a master server at 127.0.116.254:44587
I20260926 20:20:43.991976 2697 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:43.992563 2697 heartbeater.cc:511] Master 127.0.116.254:44587 requested a full tablet report, sending...
I20260926 20:20:43.993623 2213 ts_manager.cc:194] Registered new tserver with Master: bd1532df667044eeb45599ac193e9617 (127.0.116.195:41685)
I20260926 20:20:43.994372 2213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:60695
I20260926 20:20:44.000857 2703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:43.993975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44587 (local address 127.0.116.195:53635)
0926 20:20:43.994235 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:43.994250 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:43.994352 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:43.994595 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:43.994602 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:43.994819 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:43.994948 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:43.994955 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.995965 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.995968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:43.996570 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:43.996579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.997017 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.997019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:43.997090 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:43.997644 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:43.997656 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:43.997986 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:43.999545 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:43.999548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:43.999550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:43.999751 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:44.000039 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:44.000041 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:44.000043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:44.000119 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:44.000443 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:44.000445 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:44.000523 (+ 78us) client_negotiation.cc:769] Sending connection context
0926 20:20:44.000607 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:20:44.000716 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":104,"threads_started":1}
I20260926 20:20:44.000880 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:43.994028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:53635 (local address 127.0.116.254:44587)
0926 20:20:43.994157 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:43.994162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:43.994317 (+ 155us) server_negotiation.cc:407] Connection header received
0926 20:20:43.994433 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:43.994435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:43.994500 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:43.994572 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:43.995066 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.995825 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.996682 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:43.996917 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:43.996996 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:43.998101 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:43.998103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:43.998105 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:43.998122 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:43.999408 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:43.999881 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:43.999883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:43.999885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:43.999961 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:44.000195 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:44.000197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:44.000198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:44.000361 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:44.000434 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:44.000664 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 20:20:44.000755 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:20:44.004575 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:20:44 dist-test-slave-qsm9 krb5kdc[2169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:20:44 dist-test-slave-qsm9 krb5kdc[2169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:44.031587 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.022858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39878 (local address 127.0.116.254:44587)
0926 20:20:44.022986 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:44.022989 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:44.023002 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:20:44.023043 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:44.023045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:44.023092 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:44.023164 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:44.023958 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.024482 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.025360 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.025561 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.025635 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.028314 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:44.028318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:44.028320 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:44.028347 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:44.029920 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:44.030347 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:44.030349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:44.030350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:44.030429 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:44.030690 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:44.030693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:44.030695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:44.030905 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:44.031008 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:44.031258 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 20:20:44.031372 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:20:44.034700 2213 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:20:44.037760 2213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:20:44.052697 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.047083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51448 (local address 127.0.116.195:41685)
0926 20:20:44.047772 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:44.047784 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:44.048064 (+ 280us) server_negotiation.cc:407] Connection header received
0926 20:20:44.048242 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:44.048248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:44.048431 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:44.048573 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:44.048795 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.049893 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.050928 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.051845 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.051976 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.052153 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:44.052407 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 20:20:44.052525 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":87,"threads_started":1}
I20260926 20:20:44.052737 2714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.046697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:41685 (local address 127.0.0.1:51448)
0926 20:20:44.047898 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:44.047911 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:44.047990 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:44.048563 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:44.048567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:44.048584 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:44.048679 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:44.048685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.049997 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.050002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:44.050807 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:44.050816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.051967 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.051972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:44.052155 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.052243 (+ 88us) client_negotiation.cc:769] Sending connection context
0926 20:20:44.052357 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 20:20:44.052550 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1141,"mutex_wait_us":107,"thread_start_us":83,"threads_started":1}
I20260926 20:20:44.053234 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.047102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48690 (local address 127.0.116.193:35897)
0926 20:20:44.047654 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:44.047661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:44.047685 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:20:44.047759 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:44.047764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:44.048049 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:44.048210 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:44.048398 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.049619 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.051072 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.051873 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.052778 (+ 905us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.052948 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:44.052997 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 20:20:44.053087 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":73,"threads_started":1}
I20260926 20:20:44.053260 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.046455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40586 (local address 127.0.116.194:38091)
0926 20:20:44.047414 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:44.047424 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:44.047443 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:20:44.047545 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:44.047551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:44.047782 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:44.048308 (+ 526us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:44.048464 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.049425 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.051626 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.052337 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.052784 (+ 447us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.052950 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:44.053002 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 20:20:44.053093 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":102,"threads_started":1}
I20260926 20:20:44.052681 2717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.046455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:35897 (local address 127.0.0.1:48690)
0926 20:20:44.047004 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:44.047020 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:44.047121 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:44.048161 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:44.048165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:44.048180 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:44.048281 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:44.048287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.049726 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.049729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:44.050957 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:44.050967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.052005 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.052009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:44.052195 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.052241 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 20:20:44.052358 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 20:20:44.052543 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":103,"thread_start_us":104,"threads_started":1}
I20260926 20:20:44.053517 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.046291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:38091 (local address 127.0.0.1:40586)
0926 20:20:44.046935 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:44.046988 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:44.047118 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:44.047960 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:44.047968 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:44.048002 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:44.048155 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:44.048161 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.049575 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.049580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:44.051473 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:44.051481 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.052472 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.052476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:44.052559 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.052579 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 20:20:44.052629 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 20:20:44.052684 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":50,"spinlock_wait_cycles":896,"thread_start_us":96,"threads_started":1}
I20260926 20:20:44.056243 2631 tablet_service.cc:1467] Processing CreateTablet for tablet 4a25a7261ce44811b96a0d28d370b9cb (DEFAULT_TABLE table=test-table [id=eca0ef97931a45bb9ad3d1bb08c2d80f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:44.056250 2491 tablet_service.cc:1467] Processing CreateTablet for tablet 4a25a7261ce44811b96a0d28d370b9cb (DEFAULT_TABLE table=test-table [id=eca0ef97931a45bb9ad3d1bb08c2d80f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:44.057405 2631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a25a7261ce44811b96a0d28d370b9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:44.057400 2491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a25a7261ce44811b96a0d28d370b9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:44.058876 2351 tablet_service.cc:1467] Processing CreateTablet for tablet 4a25a7261ce44811b96a0d28d370b9cb (DEFAULT_TABLE table=test-table [id=eca0ef97931a45bb9ad3d1bb08c2d80f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:44.059844 2351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a25a7261ce44811b96a0d28d370b9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:44.065315 2720 tablet_bootstrap.cc:492] T 4a25a7261ce44811b96a0d28d370b9cb P 0dbe95cf940549da948f8adf2256841b: Bootstrap starting.
I20260926 20:20:44.066605 2721 tablet_bootstrap.cc:492] T 4a25a7261ce44811b96a0d28d370b9cb P d99c92f36fb3458582f7aaab81182991: Bootstrap starting.
I20260926 20:20:44.066606 2722 tablet_bootstrap.cc:492] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617: Bootstrap starting.
I20260926 20:20:44.067442 2720 tablet_bootstrap.cc:654] T 4a25a7261ce44811b96a0d28d370b9cb P 0dbe95cf940549da948f8adf2256841b: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:44.068416 2720 log.cc:824] T 4a25a7261ce44811b96a0d28d370b9cb P 0dbe95cf940549da948f8adf2256841b: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:44.068687 2722 tablet_bootstrap.cc:654] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:44.068681 2721 tablet_bootstrap.cc:654] T 4a25a7261ce44811b96a0d28d370b9cb P d99c92f36fb3458582f7aaab81182991: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:44.069540 2722 log.cc:824] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:44.069540 2721 log.cc:824] T 4a25a7261ce44811b96a0d28d370b9cb P d99c92f36fb3458582f7aaab81182991: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:44.070444 2720 tablet_bootstrap.cc:492] T 4a25a7261ce44811b96a0d28d370b9cb P 0dbe95cf940549da948f8adf2256841b: No bootstrap required, opened a new log
I20260926 20:20:44.070640 2720 ts_tablet_manager.cc:1397] T 4a25a7261ce44811b96a0d28d370b9cb P 0dbe95cf940549da948f8adf2256841b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 20:20:44.071305 2722 tablet_bootstrap.cc:492] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617: No bootstrap required, opened a new log
I20260926 20:20:44.071481 2721 tablet_bootstrap.cc:492] T 4a25a7261ce44811b96a0d28d370b9cb P d99c92f36fb3458582f7aaab81182991: No bootstrap required, opened a new log
I20260926 20:20:44.071556 2722 ts_tablet_manager.cc:1397] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 20:20:44.071681 2721 ts_tablet_manager.cc:1397] T 4a25a7261ce44811b96a0d28d370b9cb P d99c92f36fb3458582f7aaab81182991: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 20:20:44.074467 2722 raft_consensus.cc:348] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.074815 2722 raft_consensus.cc:374] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:44.074909 2722 raft_consensus.cc:729] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1532df667044eeb45599ac193e9617, State: Initialized, Role: FOLLOWER
I20260926 20:20:44.075430 2722 consensus_queue.cc:260] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.075546 2721 raft_consensus.cc:348] T 4a25a7261ce44811b96a0d28d370b9cb P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.075608 2720 raft_consensus.cc:348] T 4a25a7261ce44811b96a0d28d370b9cb P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.075821 2720 raft_consensus.cc:374] T 4a25a7261ce44811b96a0d28d370b9cb P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:44.075901 2721 raft_consensus.cc:374] T 4a25a7261ce44811b96a0d28d370b9cb P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:44.075904 2720 raft_consensus.cc:729] T 4a25a7261ce44811b96a0d28d370b9cb P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dbe95cf940549da948f8adf2256841b, State: Initialized, Role: FOLLOWER
I20260926 20:20:44.076001 2721 raft_consensus.cc:729] T 4a25a7261ce44811b96a0d28d370b9cb P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d99c92f36fb3458582f7aaab81182991, State: Initialized, Role: FOLLOWER
I20260926 20:20:44.076328 2720 consensus_queue.cc:260] T 4a25a7261ce44811b96a0d28d370b9cb P 0dbe95cf940549da948f8adf2256841b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.076366 2721 consensus_queue.cc:260] T 4a25a7261ce44811b96a0d28d370b9cb P d99c92f36fb3458582f7aaab81182991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.076619 2722 ts_tablet_manager.cc:1428] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:20:44.077008 2417 heartbeater.cc:503] Master 127.0.116.254:44587 was elected leader, sending a full tablet report...
I20260926 20:20:44.077282 2720 ts_tablet_manager.cc:1428] T 4a25a7261ce44811b96a0d28d370b9cb P 0dbe95cf940549da948f8adf2256841b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 20:20:44.077282 2721 ts_tablet_manager.cc:1428] T 4a25a7261ce44811b96a0d28d370b9cb P d99c92f36fb3458582f7aaab81182991: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:20:44.077621 2557 heartbeater.cc:503] Master 127.0.116.254:44587 was elected leader, sending a full tablet report...
I20260926 20:20:44.078022 2697 heartbeater.cc:503] Master 127.0.116.254:44587 was elected leader, sending a full tablet report...
I20260926 20:20:44.113677 2726 raft_consensus.cc:482] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:20:44.113868 2726 raft_consensus.cc:504] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.114948 2726 leader_election.cc:288] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d99c92f36fb3458582f7aaab81182991 (127.0.116.194:38091), 0dbe95cf940549da948f8adf2256841b (127.0.116.193:35897)
I20260926 20:20:44.119251 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.115259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:35897 (local address 127.0.116.195:33615)
0926 20:20:44.115485 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:44.115524 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:44.115629 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:44.115935 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:44.115940 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:44.115957 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:44.116065 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:44.116069 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.116915 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.116917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:44.118039 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:44.118050 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.118859 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.118862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:44.118946 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.118965 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 20:20:44.119026 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 20:20:44.119083 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260926 20:20:44.119282 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.115372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:33615 (local address 127.0.116.193:35897)
0926 20:20:44.115526 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:44.115531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:44.115601 (+ 70us) server_negotiation.cc:407] Connection header received
0926 20:20:44.115743 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:44.115746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:44.115814 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:44.115900 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:44.116200 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.116809 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.118173 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.118745 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.118863 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.118915 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:44.119067 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 20:20:44.119154 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:20:44.119650 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.115257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:38091 (local address 127.0.116.195:54437)
0926 20:20:44.115726 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:44.115748 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:44.115856 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:44.116137 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:44.116141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:44.116160 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:44.116515 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:44.116520 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.117323 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.117326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:44.118356 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:44.118363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.119203 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.119206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:44.119309 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.119337 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 20:20:44.119408 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 20:20:44.119486 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":65,"thread_start_us":165,"threads_started":1}
I20260926 20:20:44.119784 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.115383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:54437 (local address 127.0.116.194:38091)
0926 20:20:44.115555 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:44.115559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:44.115823 (+ 264us) server_negotiation.cc:407] Connection header received
0926 20:20:44.115983 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:44.115986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:44.116036 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:44.116104 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:44.116631 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.117220 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.118463 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.119088 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.119198 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.119241 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:44.119576 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 20:20:44.119652 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:20:44.119946 2371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a7261ce44811b96a0d28d370b9cb" candidate_uuid: "bd1532df667044eeb45599ac193e9617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dbe95cf940549da948f8adf2256841b" is_pre_election: true
I20260926 20:20:44.120290 2371 raft_consensus.cc:2454] T 4a25a7261ce44811b96a0d28d370b9cb P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd1532df667044eeb45599ac193e9617 in term 0.
I20260926 20:20:44.120347 2511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a7261ce44811b96a0d28d370b9cb" candidate_uuid: "bd1532df667044eeb45599ac193e9617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99c92f36fb3458582f7aaab81182991" is_pre_election: true
I20260926 20:20:44.120711 2511 raft_consensus.cc:2454] T 4a25a7261ce44811b96a0d28d370b9cb P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd1532df667044eeb45599ac193e9617 in term 0.
I20260926 20:20:44.120801 2582 leader_election.cc:302] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dbe95cf940549da948f8adf2256841b, bd1532df667044eeb45599ac193e9617; no voters:
I20260926 20:20:44.121102 2726 raft_consensus.cc:2781] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:20:44.121196 2726 raft_consensus.cc:482] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:20:44.121240 2726 raft_consensus.cc:3037] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:44.122463 2726 raft_consensus.cc:504] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.122849 2726 leader_election.cc:288] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617 [CANDIDATE]: Term 1 election: Requested vote from peers d99c92f36fb3458582f7aaab81182991 (127.0.116.194:38091), 0dbe95cf940549da948f8adf2256841b (127.0.116.193:35897)
I20260926 20:20:44.123157 2511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a7261ce44811b96a0d28d370b9cb" candidate_uuid: "bd1532df667044eeb45599ac193e9617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99c92f36fb3458582f7aaab81182991"
I20260926 20:20:44.123173 2371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a25a7261ce44811b96a0d28d370b9cb" candidate_uuid: "bd1532df667044eeb45599ac193e9617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dbe95cf940549da948f8adf2256841b"
I20260926 20:20:44.123312 2371 raft_consensus.cc:3037] T 4a25a7261ce44811b96a0d28d370b9cb P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:44.123313 2511 raft_consensus.cc:3037] T 4a25a7261ce44811b96a0d28d370b9cb P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:44.124361 2371 raft_consensus.cc:2454] T 4a25a7261ce44811b96a0d28d370b9cb P 0dbe95cf940549da948f8adf2256841b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd1532df667044eeb45599ac193e9617 in term 1.
I20260926 20:20:44.124647 2582 leader_election.cc:302] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dbe95cf940549da948f8adf2256841b, bd1532df667044eeb45599ac193e9617; no voters:
I20260926 20:20:44.124799 2726 raft_consensus.cc:2781] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:44.124912 2511 raft_consensus.cc:2454] T 4a25a7261ce44811b96a0d28d370b9cb P d99c92f36fb3458582f7aaab81182991 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd1532df667044eeb45599ac193e9617 in term 1.
I20260926 20:20:44.125054 2726 raft_consensus.cc:686] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617 [term 1 LEADER]: Becoming Leader. State: Replica: bd1532df667044eeb45599ac193e9617, State: Running, Role: LEADER
I20260926 20:20:44.125735 2726 consensus_queue.cc:237] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.128798 2213 catalog_manager.cc:5495] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd1532df667044eeb45599ac193e9617 (127.0.116.195). New cstate: current_term: 1 leader_uuid: "bd1532df667044eeb45599ac193e9617" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } health_report { overall_health: HEALTHY } } }
Sep 26 20:20:44 dist-test-slave-qsm9 krb5kdc[2169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:44.177747 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.169382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39894 (local address 127.0.116.254:44587)
0926 20:20:44.169517 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:44.169522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:44.169691 (+ 169us) server_negotiation.cc:407] Connection header received
0926 20:20:44.169862 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:44.169865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:44.169898 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:44.169980 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:44.170625 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.171144 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.172002 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.172209 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.172314 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.174553 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:44.174556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:44.174558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:44.174578 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:44.176024 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:44.176476 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:44.176482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:44.176484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:44.176559 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:44.176827 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:44.176865 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:44.176867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:44.177034 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:44.177141 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:44.177429 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 20:20:44.177562 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":48}
I20260926 20:20:44.177758 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.169296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44587 (local address 127.0.0.1:39894)
0926 20:20:44.169604 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:44.169615 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:44.169755 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:44.169997 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:44.170000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:44.170422 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:44.170511 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:44.170517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.171249 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.171255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:44.171890 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:44.171899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.172318 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.172321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:44.172420 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.172989 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:44.173001 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:44.174405 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:44.176125 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:44.176137 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:44.176144 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:44.176378 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:44.176648 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:44.176652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:44.176655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:44.176734 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:44.177141 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:44.177149 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:44.177266 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 20:20:44.177384 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 20:20:44.177629 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":92,"thread_start_us":84,"threads_started":1}
I20260926 20:20:44.181818 2213 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:39894:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 20:20:44.182526 2213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:20:44.187541 2631 tablet_service.cc:1467] Processing CreateTablet for tablet a57d12ab484a4adc842e213f53ad0ce5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=509f5c102d2c4b9a9087f27e03f48cac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:20:44.187425 2351 tablet_service.cc:1467] Processing CreateTablet for tablet a57d12ab484a4adc842e213f53ad0ce5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=509f5c102d2c4b9a9087f27e03f48cac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:20:44.187425 2491 tablet_service.cc:1467] Processing CreateTablet for tablet a57d12ab484a4adc842e213f53ad0ce5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=509f5c102d2c4b9a9087f27e03f48cac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:20:44.188186 2631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a57d12ab484a4adc842e213f53ad0ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:44.188180 2491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a57d12ab484a4adc842e213f53ad0ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:44.188180 2351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a57d12ab484a4adc842e213f53ad0ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:44.190549 2722 tablet_bootstrap.cc:492] T a57d12ab484a4adc842e213f53ad0ce5 P bd1532df667044eeb45599ac193e9617: Bootstrap starting.
I20260926 20:20:44.190980 2721 tablet_bootstrap.cc:492] T a57d12ab484a4adc842e213f53ad0ce5 P d99c92f36fb3458582f7aaab81182991: Bootstrap starting.
I20260926 20:20:44.191483 2720 tablet_bootstrap.cc:492] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b: Bootstrap starting.
I20260926 20:20:44.191596 2722 tablet_bootstrap.cc:654] T a57d12ab484a4adc842e213f53ad0ce5 P bd1532df667044eeb45599ac193e9617: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:44.192029 2721 tablet_bootstrap.cc:654] T a57d12ab484a4adc842e213f53ad0ce5 P d99c92f36fb3458582f7aaab81182991: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:44.192521 2720 tablet_bootstrap.cc:654] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:44.193013 2722 tablet_bootstrap.cc:492] T a57d12ab484a4adc842e213f53ad0ce5 P bd1532df667044eeb45599ac193e9617: No bootstrap required, opened a new log
I20260926 20:20:44.193106 2722 ts_tablet_manager.cc:1397] T a57d12ab484a4adc842e213f53ad0ce5 P bd1532df667044eeb45599ac193e9617: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260926 20:20:44.193655 2722 raft_consensus.cc:348] T a57d12ab484a4adc842e213f53ad0ce5 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.193789 2722 raft_consensus.cc:374] T a57d12ab484a4adc842e213f53ad0ce5 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:44.193833 2722 raft_consensus.cc:729] T a57d12ab484a4adc842e213f53ad0ce5 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1532df667044eeb45599ac193e9617, State: Initialized, Role: FOLLOWER
I20260926 20:20:44.194062 2720 tablet_bootstrap.cc:492] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b: No bootstrap required, opened a new log
I20260926 20:20:44.193997 2722 consensus_queue.cc:260] T a57d12ab484a4adc842e213f53ad0ce5 P bd1532df667044eeb45599ac193e9617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.194140 2720 ts_tablet_manager.cc:1397] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:20:44.194351 2722 ts_tablet_manager.cc:1428] T a57d12ab484a4adc842e213f53ad0ce5 P bd1532df667044eeb45599ac193e9617: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:20:44.194571 2720 raft_consensus.cc:348] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.194679 2720 raft_consensus.cc:374] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:44.194710 2720 raft_consensus.cc:729] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dbe95cf940549da948f8adf2256841b, State: Initialized, Role: FOLLOWER
I20260926 20:20:44.194820 2720 consensus_queue.cc:260] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.195084 2720 ts_tablet_manager.cc:1428] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:20:44.195284 2727 tablet_replica.cc:439] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } } }
I20260926 20:20:44.195372 2721 tablet_bootstrap.cc:492] T a57d12ab484a4adc842e213f53ad0ce5 P d99c92f36fb3458582f7aaab81182991: No bootstrap required, opened a new log
I20260926 20:20:44.195413 2727 tablet_replica.cc:442] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:20:44.195454 2721 ts_tablet_manager.cc:1397] T a57d12ab484a4adc842e213f53ad0ce5 P d99c92f36fb3458582f7aaab81182991: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 20:20:44.194283 2726 tablet_replica.cc:439] T a57d12ab484a4adc842e213f53ad0ce5 P bd1532df667044eeb45599ac193e9617: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } } }
I20260926 20:20:44.195888 2721 raft_consensus.cc:348] T a57d12ab484a4adc842e213f53ad0ce5 P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.195999 2721 raft_consensus.cc:374] T a57d12ab484a4adc842e213f53ad0ce5 P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:44.195996 2726 tablet_replica.cc:442] T a57d12ab484a4adc842e213f53ad0ce5 P bd1532df667044eeb45599ac193e9617: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:20:44.196034 2721 raft_consensus.cc:729] T a57d12ab484a4adc842e213f53ad0ce5 P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d99c92f36fb3458582f7aaab81182991, State: Initialized, Role: FOLLOWER
I20260926 20:20:44.196154 2721 consensus_queue.cc:260] T a57d12ab484a4adc842e213f53ad0ce5 P d99c92f36fb3458582f7aaab81182991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.196416 2728 tablet_replica.cc:439] T a57d12ab484a4adc842e213f53ad0ce5 P d99c92f36fb3458582f7aaab81182991: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } } }
I20260926 20:20:44.196544 2728 tablet_replica.cc:442] T a57d12ab484a4adc842e213f53ad0ce5 P d99c92f36fb3458582f7aaab81182991: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:20:44.196846 2721 ts_tablet_manager.cc:1428] T a57d12ab484a4adc842e213f53ad0ce5 P d99c92f36fb3458582f7aaab81182991: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260926 20:20:44.237967 2698 tablet.cc:2369] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:20:44.240653 2558 tablet.cc:2369] T 4a25a7261ce44811b96a0d28d370b9cb P d99c92f36fb3458582f7aaab81182991: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:20:44.255877 2418 tablet.cc:2369] T 4a25a7261ce44811b96a0d28d370b9cb P 0dbe95cf940549da948f8adf2256841b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:20:44.313515 2727 raft_consensus.cc:482] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:20:44.313643 2727 raft_consensus.cc:504] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.314713 2727 leader_election.cc:288] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d99c92f36fb3458582f7aaab81182991 (127.0.116.194:38091), bd1532df667044eeb45599ac193e9617 (127.0.116.195:41685)
I20260926 20:20:44.319094 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.315027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:59579 (local address 127.0.116.194:38091)
0926 20:20:44.315163 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:44.315168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:44.315421 (+ 253us) server_negotiation.cc:407] Connection header received
0926 20:20:44.315631 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:44.315633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:44.315674 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:44.315732 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:44.316043 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.316795 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.317997 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.318565 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.318653 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.318693 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:44.318918 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 20:20:44.318972 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:20:44.319150 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.314928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:38091 (local address 127.0.116.193:59579)
0926 20:20:44.315313 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:44.315325 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:44.315455 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:44.315772 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:44.315775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:44.315790 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:44.315935 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:44.315940 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.316900 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.316904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:44.317882 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:44.317890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.318691 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.318695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:44.318794 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.318820 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 20:20:44.318896 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 20:20:44.318961 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"spinlock_wait_cycles":28544,"thread_start_us":153,"threads_started":1}
I20260926 20:20:44.319700 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.315075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:36259 (local address 127.0.116.195:41685)
0926 20:20:44.315217 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:44.315223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:44.315467 (+ 244us) server_negotiation.cc:407] Connection header received
0926 20:20:44.315641 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:44.315644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:44.315687 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:44.315777 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:44.316657 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.317342 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.318462 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.319012 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.319116 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.319173 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:44.319536 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 20:20:44.319616 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:20:44.319720 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.314982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:41685 (local address 127.0.116.193:36259)
0926 20:20:44.315330 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:44.315356 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:44.315479 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:44.315805 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:44.315808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:44.315850 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:44.316540 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:44.316549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.317451 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.317454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:44.318346 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:44.318356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.319131 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.319137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:44.319405 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.319427 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:20:44.319516 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 20:20:44.319581 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"spinlock_wait_cycles":3200,"thread_start_us":104,"threads_started":1}
I20260926 20:20:44.319674 2511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a57d12ab484a4adc842e213f53ad0ce5" candidate_uuid: "0dbe95cf940549da948f8adf2256841b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99c92f36fb3458582f7aaab81182991" is_pre_election: true
I20260926 20:20:44.319816 2511 raft_consensus.cc:2454] T a57d12ab484a4adc842e213f53ad0ce5 P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dbe95cf940549da948f8adf2256841b in term 0.
I20260926 20:20:44.320199 2305 leader_election.cc:302] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dbe95cf940549da948f8adf2256841b, d99c92f36fb3458582f7aaab81182991; no voters:
I20260926 20:20:44.320478 2727 raft_consensus.cc:2781] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:20:44.320442 2637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a57d12ab484a4adc842e213f53ad0ce5" candidate_uuid: "0dbe95cf940549da948f8adf2256841b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd1532df667044eeb45599ac193e9617" is_pre_election: true
I20260926 20:20:44.320545 2727 raft_consensus.cc:482] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:20:44.320574 2727 raft_consensus.cc:3037] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:44.320667 2637 raft_consensus.cc:2454] T a57d12ab484a4adc842e213f53ad0ce5 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dbe95cf940549da948f8adf2256841b in term 0.
I20260926 20:20:44.321862 2727 raft_consensus.cc:504] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.322229 2727 leader_election.cc:288] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b [CANDIDATE]: Term 1 election: Requested vote from peers d99c92f36fb3458582f7aaab81182991 (127.0.116.194:38091), bd1532df667044eeb45599ac193e9617 (127.0.116.195:41685)
I20260926 20:20:44.322535 2511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a57d12ab484a4adc842e213f53ad0ce5" candidate_uuid: "0dbe95cf940549da948f8adf2256841b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99c92f36fb3458582f7aaab81182991"
I20260926 20:20:44.322592 2637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a57d12ab484a4adc842e213f53ad0ce5" candidate_uuid: "0dbe95cf940549da948f8adf2256841b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd1532df667044eeb45599ac193e9617"
I20260926 20:20:44.322651 2511 raft_consensus.cc:3037] T a57d12ab484a4adc842e213f53ad0ce5 P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:44.322680 2637 raft_consensus.cc:3037] T a57d12ab484a4adc842e213f53ad0ce5 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:44.323805 2637 raft_consensus.cc:2454] T a57d12ab484a4adc842e213f53ad0ce5 P bd1532df667044eeb45599ac193e9617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dbe95cf940549da948f8adf2256841b in term 1.
I20260926 20:20:44.323805 2511 raft_consensus.cc:2454] T a57d12ab484a4adc842e213f53ad0ce5 P d99c92f36fb3458582f7aaab81182991 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dbe95cf940549da948f8adf2256841b in term 1.
I20260926 20:20:44.324061 2305 leader_election.cc:302] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dbe95cf940549da948f8adf2256841b, d99c92f36fb3458582f7aaab81182991; no voters:
I20260926 20:20:44.324285 2727 raft_consensus.cc:2781] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:44.324455 2727 raft_consensus.cc:686] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b [term 1 LEADER]: Becoming Leader. State: Replica: 0dbe95cf940549da948f8adf2256841b, State: Running, Role: LEADER
I20260926 20:20:44.324806 2727 consensus_queue.cc:237] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.327157 2743 tablet_replica.cc:439] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b: TxnStatusTablet state changed. Reason: New leader 0dbe95cf940549da948f8adf2256841b. Latest consensus state: current_term: 1 leader_uuid: "0dbe95cf940549da948f8adf2256841b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } } }
I20260926 20:20:44.327260 2743 tablet_replica.cc:442] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:20:44.327566 2745 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 20:20:44.327781 2745 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 20:20:44.328418 2213 catalog_manager.cc:5495] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dbe95cf940549da948f8adf2256841b (127.0.116.193). New cstate: current_term: 1 leader_uuid: "0dbe95cf940549da948f8adf2256841b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } health_report { overall_health: UNKNOWN } } }
I20260926 20:20:44.352959 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.349159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48696 (local address 127.0.116.193:35897)
0926 20:20:44.349266 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:44.349269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:44.349384 (+ 115us) server_negotiation.cc:407] Connection header received
0926 20:20:44.349569 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:44.349571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:44.349623 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:44.349702 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:20:44.349936 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.350415 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.351600 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.351806 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.351890 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.352194 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:20:44.352592 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:20:44.352673 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:44.352796 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 20:20:44.352851 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 20:20:44.352993 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.349078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:35897 (local address 127.0.0.1:48696)
0926 20:20:44.349314 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:44.349329 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:44.349448 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:44.349737 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:44.349741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:44.349748 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:20:44.349856 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:44.349862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.350509 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.350512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:44.351474 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:44.351484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.351914 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.351917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:44.352009 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.352061 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:20:44.352680 (+ 619us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:20:44.352695 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 20:20:44.352765 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 20:20:44.352845 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260926 20:20:44.358299 2511 raft_consensus.cc:1261] T a57d12ab484a4adc842e213f53ad0ce5 P d99c92f36fb3458582f7aaab81182991 [term 1 FOLLOWER]: Refusing update from remote peer 0dbe95cf940549da948f8adf2256841b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:44.358297 2637 raft_consensus.cc:1261] T a57d12ab484a4adc842e213f53ad0ce5 P bd1532df667044eeb45599ac193e9617 [term 1 FOLLOWER]: Refusing update from remote peer 0dbe95cf940549da948f8adf2256841b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:44.358959 2743 consensus_queue.cc:1035] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:20:44.359169 2727 consensus_queue.cc:1035] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:20:44.362322 2726 tablet_replica.cc:439] T a57d12ab484a4adc842e213f53ad0ce5 P bd1532df667044eeb45599ac193e9617: TxnStatusTablet state changed. Reason: New leader 0dbe95cf940549da948f8adf2256841b. Latest consensus state: current_term: 1 leader_uuid: "0dbe95cf940549da948f8adf2256841b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } } }
I20260926 20:20:44.362442 2726 tablet_replica.cc:442] T a57d12ab484a4adc842e213f53ad0ce5 P bd1532df667044eeb45599ac193e9617: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:20:44.364048 2728 tablet_replica.cc:439] T a57d12ab484a4adc842e213f53ad0ce5 P d99c92f36fb3458582f7aaab81182991: TxnStatusTablet state changed. Reason: New leader 0dbe95cf940549da948f8adf2256841b. Latest consensus state: current_term: 1 leader_uuid: "0dbe95cf940549da948f8adf2256841b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } } }
I20260926 20:20:44.364207 2728 tablet_replica.cc:442] T a57d12ab484a4adc842e213f53ad0ce5 P d99c92f36fb3458582f7aaab81182991: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:20:44.366312 2727 tablet_replica.cc:439] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0dbe95cf940549da948f8adf2256841b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } } }
I20260926 20:20:44.366452 2727 tablet_replica.cc:442] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:20:44.366575 2727 tablet_replica.cc:439] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0dbe95cf940549da948f8adf2256841b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } } }
I20260926 20:20:44.366642 2727 tablet_replica.cc:442] T a57d12ab484a4adc842e213f53ad0ce5 P 0dbe95cf940549da948f8adf2256841b: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:20:44.366707 2728 tablet_replica.cc:439] T a57d12ab484a4adc842e213f53ad0ce5 P d99c92f36fb3458582f7aaab81182991: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0dbe95cf940549da948f8adf2256841b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } } }
I20260926 20:20:44.366806 2728 tablet_replica.cc:442] T a57d12ab484a4adc842e213f53ad0ce5 P d99c92f36fb3458582f7aaab81182991: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:20:44.367107 2749 mvcc.cc:205] Tried to move back new op lower bound from 7333699765684633600 to 7333699765556817920. Current Snapshot: MvccSnapshot[applied={T|T < 7333699765684633600}]
I20260926 20:20:44.367411 2748 mvcc.cc:205] Tried to move back new op lower bound from 7333699765684633600 to 7333699765556817920. Current Snapshot: MvccSnapshot[applied={T|T < 7333699765684633600}]
I20260926 20:20:44.367548 2750 mvcc.cc:205] Tried to move back new op lower bound from 7333699765684633600 to 7333699765556817920. Current Snapshot: MvccSnapshot[applied={T|T < 7333699765684633600}]
I20260926 20:20:44.368468 2726 tablet_replica.cc:439] T a57d12ab484a4adc842e213f53ad0ce5 P bd1532df667044eeb45599ac193e9617: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0dbe95cf940549da948f8adf2256841b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } } }
I20260926 20:20:44.368577 2726 tablet_replica.cc:442] T a57d12ab484a4adc842e213f53ad0ce5 P bd1532df667044eeb45599ac193e9617: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:20:44.378358 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.374464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.0.116.195:41685)
0926 20:20:44.374583 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:44.374588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:44.374603 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:20:44.374770 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:44.374773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:44.374832 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:44.374934 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:20:44.375135 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.375936 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.376734 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.377069 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.377167 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.377398 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:20:44.377898 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:20:44.378009 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:44.378138 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 20:20:44.378196 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:20:44.389189 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.385831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:55957 (local address 127.0.116.193:35897)
0926 20:20:44.385951 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:44.385954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:44.385966 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:20:44.386074 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:44.386077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:44.386116 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:44.386176 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:20:44.386436 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.386945 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.388165 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.388372 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.388446 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.388789 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:20:44.388866 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:20:44.388927 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:44.389041 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 20:20:44.389086 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:20:44.389222 2703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.385693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:35897 (local address 127.0.116.195:55957)
0926 20:20:44.385907 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:44.385922 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:44.385996 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:44.386202 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:44.386205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:44.386212 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:20:44.386320 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:44.386325 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.387044 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.387047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:44.388059 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:44.388068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.388470 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.388474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:44.388564 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.388615 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:20:44.388951 (+ 336us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:20:44.388957 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:20:44.389022 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 20:20:44.389093 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260926 20:20:44.396029 2511 raft_consensus.cc:1261] T 4a25a7261ce44811b96a0d28d370b9cb P d99c92f36fb3458582f7aaab81182991 [term 1 FOLLOWER]: Refusing update from remote peer bd1532df667044eeb45599ac193e9617: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:44.396117 2371 raft_consensus.cc:1261] T 4a25a7261ce44811b96a0d28d370b9cb P 0dbe95cf940549da948f8adf2256841b [term 1 FOLLOWER]: Refusing update from remote peer bd1532df667044eeb45599ac193e9617: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:44.396554 2726 consensus_queue.cc:1035] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:20:44.396750 2730 consensus_queue.cc:1035] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:20:44.420583 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.416472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:41685 (local address 127.0.116.193:48487)
0926 20:20:44.416833 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:44.416848 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:44.416935 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:44.417150 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:44.417152 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:44.417156 (+ 4us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:20:44.417288 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:44.417294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.418312 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.418315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:44.419194 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:44.419203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.419867 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.419870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:44.419937 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.419985 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:20:44.420292 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:20:44.420299 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 20:20:44.420368 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:20:44.420455 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":131,"threads_started":1}
I20260926 20:20:44.420593 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:44.416533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:48487 (local address 127.0.116.195:41685)
0926 20:20:44.416668 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:44.416673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:44.416919 (+ 246us) server_negotiation.cc:407] Connection header received
0926 20:20:44.417006 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:44.417009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:44.417061 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:44.417145 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:20:44.417405 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.418199 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.419304 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:44.419703 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:44.419799 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:44.420104 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:20:44.420181 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:20:44.420251 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:44.420392 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 20:20:44.420443 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:20:44.448310 2213 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:20:44.448561 2213 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:20:44.451181 2213 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972: Sending DeleteTablet for 3 replicas of tablet 4a25a7261ce44811b96a0d28d370b9cb
I20260926 20:20:44.452049 2491 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a25a7261ce44811b96a0d28d370b9cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:44 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:40586
I20260926 20:20:44.452548 2631 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a25a7261ce44811b96a0d28d370b9cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:44 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:51448
I20260926 20:20:44.452549 2351 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a25a7261ce44811b96a0d28d370b9cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:44 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:48690
I20260926 20:20:44.453016 2768 tablet_replica.cc:330] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617: stopping tablet replica
I20260926 20:20:44.453356 2768 raft_consensus.cc:2229] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:20:44.453785 2768 raft_consensus.cc:2258] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:20:44.453764 2213 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:20:44.454209 2213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:20:44.455050 2769 tablet_replica.cc:330] T 4a25a7261ce44811b96a0d28d370b9cb P 0dbe95cf940549da948f8adf2256841b: stopping tablet replica
I20260926 20:20:44.455413 2769 raft_consensus.cc:2229] T 4a25a7261ce44811b96a0d28d370b9cb P 0dbe95cf940549da948f8adf2256841b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:20:44.455672 2769 raft_consensus.cc:2258] T 4a25a7261ce44811b96a0d28d370b9cb P 0dbe95cf940549da948f8adf2256841b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:20:44.455675 2768 ts_tablet_manager.cc:1905] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:20:44.458065 2769 ts_tablet_manager.cc:1905] T 4a25a7261ce44811b96a0d28d370b9cb P 0dbe95cf940549da948f8adf2256841b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:20:44.458799 2768 ts_tablet_manager.cc:1918] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:20:44.458910 2768 log.cc:1196] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/4a25a7261ce44811b96a0d28d370b9cb
I20260926 20:20:44.459220 2768 ts_tablet_manager.cc:1939] T 4a25a7261ce44811b96a0d28d370b9cb P bd1532df667044eeb45599ac193e9617: Deleting consensus metadata
I20260926 20:20:44.459693 2767 tablet_replica.cc:330] T 4a25a7261ce44811b96a0d28d370b9cb P d99c92f36fb3458582f7aaab81182991: stopping tablet replica
I20260926 20:20:44.460011 2767 raft_consensus.cc:2229] T 4a25a7261ce44811b96a0d28d370b9cb P d99c92f36fb3458582f7aaab81182991 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:20:44.460103 2351 tablet_service.cc:1467] Processing CreateTablet for tablet 583df4b8cc034a4eb335f0dccf4bf222 (DEFAULT_TABLE table=test-table [id=45861cf3af72429a8d7c862347dc1917]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:44.460103 2491 tablet_service.cc:1467] Processing CreateTablet for tablet 583df4b8cc034a4eb335f0dccf4bf222 (DEFAULT_TABLE table=test-table [id=45861cf3af72429a8d7c862347dc1917]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:44.460223 2767 raft_consensus.cc:2258] T 4a25a7261ce44811b96a0d28d370b9cb P d99c92f36fb3458582f7aaab81182991 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:20:44.460377 2351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 583df4b8cc034a4eb335f0dccf4bf222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:44.460389 2491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 583df4b8cc034a4eb335f0dccf4bf222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:44.461696 2769 ts_tablet_manager.cc:1918] T 4a25a7261ce44811b96a0d28d370b9cb P 0dbe95cf940549da948f8adf2256841b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:20:44.461843 2769 log.cc:1196] T 4a25a7261ce44811b96a0d28d370b9cb P 0dbe95cf940549da948f8adf2256841b: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-0/wal/wals/4a25a7261ce44811b96a0d28d370b9cb
I20260926 20:20:44.462211 2769 ts_tablet_manager.cc:1939] T 4a25a7261ce44811b96a0d28d370b9cb P 0dbe95cf940549da948f8adf2256841b: Deleting consensus metadata
I20260926 20:20:44.462775 2720 tablet_bootstrap.cc:492] T 583df4b8cc034a4eb335f0dccf4bf222 P 0dbe95cf940549da948f8adf2256841b: Bootstrap starting.
I20260926 20:20:44.462952 2721 tablet_bootstrap.cc:492] T 583df4b8cc034a4eb335f0dccf4bf222 P d99c92f36fb3458582f7aaab81182991: Bootstrap starting.
I20260926 20:20:44.463757 2720 tablet_bootstrap.cc:654] T 583df4b8cc034a4eb335f0dccf4bf222 P 0dbe95cf940549da948f8adf2256841b: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:44.463863 2721 tablet_bootstrap.cc:654] T 583df4b8cc034a4eb335f0dccf4bf222 P d99c92f36fb3458582f7aaab81182991: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:44.464974 2198 catalog_manager.cc:4841] TS 0dbe95cf940549da948f8adf2256841b (127.0.116.193:35897): tablet 4a25a7261ce44811b96a0d28d370b9cb (table test-table [id=eca0ef97931a45bb9ad3d1bb08c2d80f]) successfully deleted
I20260926 20:20:44.465356 2767 ts_tablet_manager.cc:1905] T 4a25a7261ce44811b96a0d28d370b9cb P d99c92f36fb3458582f7aaab81182991: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:20:44.465337 2200 catalog_manager.cc:4841] TS bd1532df667044eeb45599ac193e9617 (127.0.116.195:41685): tablet 4a25a7261ce44811b96a0d28d370b9cb (table test-table [id=eca0ef97931a45bb9ad3d1bb08c2d80f]) successfully deleted
I20260926 20:20:44.465714 2721 tablet_bootstrap.cc:492] T 583df4b8cc034a4eb335f0dccf4bf222 P d99c92f36fb3458582f7aaab81182991: No bootstrap required, opened a new log
I20260926 20:20:44.465791 2721 ts_tablet_manager.cc:1397] T 583df4b8cc034a4eb335f0dccf4bf222 P d99c92f36fb3458582f7aaab81182991: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 20:20:44.465853 2631 tablet_service.cc:1467] Processing CreateTablet for tablet 583df4b8cc034a4eb335f0dccf4bf222 (DEFAULT_TABLE table=test-table [id=45861cf3af72429a8d7c862347dc1917]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:44.466085 2631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 583df4b8cc034a4eb335f0dccf4bf222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:44.466223 2721 raft_consensus.cc:348] T 583df4b8cc034a4eb335f0dccf4bf222 P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.466346 2721 raft_consensus.cc:374] T 583df4b8cc034a4eb335f0dccf4bf222 P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:44.466382 2721 raft_consensus.cc:729] T 583df4b8cc034a4eb335f0dccf4bf222 P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d99c92f36fb3458582f7aaab81182991, State: Initialized, Role: FOLLOWER
I20260926 20:20:44.466524 2721 consensus_queue.cc:260] T 583df4b8cc034a4eb335f0dccf4bf222 P d99c92f36fb3458582f7aaab81182991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.466701 2720 tablet_bootstrap.cc:492] T 583df4b8cc034a4eb335f0dccf4bf222 P 0dbe95cf940549da948f8adf2256841b: No bootstrap required, opened a new log
I20260926 20:20:44.466754 2721 ts_tablet_manager.cc:1428] T 583df4b8cc034a4eb335f0dccf4bf222 P d99c92f36fb3458582f7aaab81182991: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:20:44.466773 2720 ts_tablet_manager.cc:1397] T 583df4b8cc034a4eb335f0dccf4bf222 P 0dbe95cf940549da948f8adf2256841b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 20:20:44.467214 2720 raft_consensus.cc:348] T 583df4b8cc034a4eb335f0dccf4bf222 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.467314 2720 raft_consensus.cc:374] T 583df4b8cc034a4eb335f0dccf4bf222 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:44.467360 2720 raft_consensus.cc:729] T 583df4b8cc034a4eb335f0dccf4bf222 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dbe95cf940549da948f8adf2256841b, State: Initialized, Role: FOLLOWER
I20260926 20:20:44.467473 2720 consensus_queue.cc:260] T 583df4b8cc034a4eb335f0dccf4bf222 P 0dbe95cf940549da948f8adf2256841b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.467728 2720 ts_tablet_manager.cc:1428] T 583df4b8cc034a4eb335f0dccf4bf222 P 0dbe95cf940549da948f8adf2256841b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:20:44.468158 2767 ts_tablet_manager.cc:1918] T 4a25a7261ce44811b96a0d28d370b9cb P d99c92f36fb3458582f7aaab81182991: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:20:44.468241 2767 log.cc:1196] T 4a25a7261ce44811b96a0d28d370b9cb P d99c92f36fb3458582f7aaab81182991: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/4a25a7261ce44811b96a0d28d370b9cb
I20260926 20:20:44.468549 2767 ts_tablet_manager.cc:1939] T 4a25a7261ce44811b96a0d28d370b9cb P d99c92f36fb3458582f7aaab81182991: Deleting consensus metadata
I20260926 20:20:44.468758 2722 tablet_bootstrap.cc:492] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617: Bootstrap starting.
I20260926 20:20:44.469756 2722 tablet_bootstrap.cc:654] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:44.471235 2722 tablet_bootstrap.cc:492] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617: No bootstrap required, opened a new log
I20260926 20:20:44.471351 2722 ts_tablet_manager.cc:1397] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260926 20:20:44.471797 2722 raft_consensus.cc:348] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.471890 2722 raft_consensus.cc:374] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:44.471911 2722 raft_consensus.cc:729] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1532df667044eeb45599ac193e9617, State: Initialized, Role: FOLLOWER
I20260926 20:20:44.472040 2722 consensus_queue.cc:260] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.472301 2722 ts_tablet_manager.cc:1428] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:20:44.480069 2201 catalog_manager.cc:4841] TS d99c92f36fb3458582f7aaab81182991 (127.0.116.194:38091): tablet 4a25a7261ce44811b96a0d28d370b9cb (table test-table [id=eca0ef97931a45bb9ad3d1bb08c2d80f]) successfully deleted
I20260926 20:20:44.488830 2730 raft_consensus.cc:482] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:20:44.488996 2730 raft_consensus.cc:504] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.489425 2730 leader_election.cc:288] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dbe95cf940549da948f8adf2256841b (127.0.116.193:35897), d99c92f36fb3458582f7aaab81182991 (127.0.116.194:38091)
I20260926 20:20:44.490005 2371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583df4b8cc034a4eb335f0dccf4bf222" candidate_uuid: "bd1532df667044eeb45599ac193e9617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dbe95cf940549da948f8adf2256841b" is_pre_election: true
I20260926 20:20:44.490163 2371 raft_consensus.cc:2454] T 583df4b8cc034a4eb335f0dccf4bf222 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd1532df667044eeb45599ac193e9617 in term 0.
I20260926 20:20:44.490470 2582 leader_election.cc:302] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dbe95cf940549da948f8adf2256841b, bd1532df667044eeb45599ac193e9617; no voters:
I20260926 20:20:44.490567 2510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583df4b8cc034a4eb335f0dccf4bf222" candidate_uuid: "bd1532df667044eeb45599ac193e9617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99c92f36fb3458582f7aaab81182991" is_pre_election: true
I20260926 20:20:44.490634 2730 raft_consensus.cc:2781] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:20:44.490675 2730 raft_consensus.cc:482] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:20:44.490715 2730 raft_consensus.cc:3037] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:44.490702 2510 raft_consensus.cc:2454] T 583df4b8cc034a4eb335f0dccf4bf222 P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd1532df667044eeb45599ac193e9617 in term 0.
I20260926 20:20:44.492093 2730 raft_consensus.cc:504] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.492481 2730 leader_election.cc:288] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617 [CANDIDATE]: Term 1 election: Requested vote from peers 0dbe95cf940549da948f8adf2256841b (127.0.116.193:35897), d99c92f36fb3458582f7aaab81182991 (127.0.116.194:38091)
I20260926 20:20:44.492817 2371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583df4b8cc034a4eb335f0dccf4bf222" candidate_uuid: "bd1532df667044eeb45599ac193e9617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dbe95cf940549da948f8adf2256841b"
I20260926 20:20:44.492856 2510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583df4b8cc034a4eb335f0dccf4bf222" candidate_uuid: "bd1532df667044eeb45599ac193e9617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99c92f36fb3458582f7aaab81182991"
I20260926 20:20:44.492956 2371 raft_consensus.cc:3037] T 583df4b8cc034a4eb335f0dccf4bf222 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:44.492980 2510 raft_consensus.cc:3037] T 583df4b8cc034a4eb335f0dccf4bf222 P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:44.494074 2510 raft_consensus.cc:2454] T 583df4b8cc034a4eb335f0dccf4bf222 P d99c92f36fb3458582f7aaab81182991 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd1532df667044eeb45599ac193e9617 in term 1.
I20260926 20:20:44.494074 2371 raft_consensus.cc:2454] T 583df4b8cc034a4eb335f0dccf4bf222 P 0dbe95cf940549da948f8adf2256841b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd1532df667044eeb45599ac193e9617 in term 1.
I20260926 20:20:44.494385 2585 leader_election.cc:302] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0dbe95cf940549da948f8adf2256841b, bd1532df667044eeb45599ac193e9617, d99c92f36fb3458582f7aaab81182991; no voters:
I20260926 20:20:44.494571 2730 raft_consensus.cc:2781] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:44.494712 2730 raft_consensus.cc:686] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617 [term 1 LEADER]: Becoming Leader. State: Replica: bd1532df667044eeb45599ac193e9617, State: Running, Role: LEADER
I20260926 20:20:44.494863 2730 consensus_queue.cc:237] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:44.496613 2212 catalog_manager.cc:5495] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd1532df667044eeb45599ac193e9617 (127.0.116.195). New cstate: current_term: 1 leader_uuid: "bd1532df667044eeb45599ac193e9617" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } health_report { overall_health: HEALTHY } } }
I20260926 20:20:44.519814 2371 raft_consensus.cc:1261] T 583df4b8cc034a4eb335f0dccf4bf222 P 0dbe95cf940549da948f8adf2256841b [term 1 FOLLOWER]: Refusing update from remote peer bd1532df667044eeb45599ac193e9617: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:44.520197 2726 consensus_queue.cc:1035] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:20:44.523345 2510 raft_consensus.cc:1261] T 583df4b8cc034a4eb335f0dccf4bf222 P d99c92f36fb3458582f7aaab81182991 [term 1 FOLLOWER]: Refusing update from remote peer bd1532df667044eeb45599ac193e9617: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:44.525436 2726 consensus_queue.cc:1035] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 20:20:44.542431 2749 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 20:20:46.092130 2212 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:20:46.092329 2212 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:20:46.094662 2212 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972: Sending DeleteTablet for 3 replicas of tablet 583df4b8cc034a4eb335f0dccf4bf222
I20260926 20:20:46.095253 2491 tablet_service.cc:1514] Processing DeleteTablet for tablet 583df4b8cc034a4eb335f0dccf4bf222 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:46 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:40586
I20260926 20:20:46.095403 2631 tablet_service.cc:1514] Processing DeleteTablet for tablet 583df4b8cc034a4eb335f0dccf4bf222 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:46 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:51448
I20260926 20:20:46.095606 2351 tablet_service.cc:1514] Processing DeleteTablet for tablet 583df4b8cc034a4eb335f0dccf4bf222 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:46 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:48690
I20260926 20:20:46.095721 2797 tablet_replica.cc:330] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617: stopping tablet replica
I20260926 20:20:46.095883 2797 raft_consensus.cc:2229] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:20:46.096009 2798 tablet_replica.cc:330] T 583df4b8cc034a4eb335f0dccf4bf222 P 0dbe95cf940549da948f8adf2256841b: stopping tablet replica
I20260926 20:20:46.096112 2797 raft_consensus.cc:2258] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:20:46.096164 2798 raft_consensus.cc:2229] T 583df4b8cc034a4eb335f0dccf4bf222 P 0dbe95cf940549da948f8adf2256841b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:20:46.096287 2798 raft_consensus.cc:2258] T 583df4b8cc034a4eb335f0dccf4bf222 P 0dbe95cf940549da948f8adf2256841b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:20:46.096529 2796 tablet_replica.cc:330] T 583df4b8cc034a4eb335f0dccf4bf222 P d99c92f36fb3458582f7aaab81182991: stopping tablet replica
I20260926 20:20:46.096715 2796 raft_consensus.cc:2229] T 583df4b8cc034a4eb335f0dccf4bf222 P d99c92f36fb3458582f7aaab81182991 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:20:46.096840 2796 raft_consensus.cc:2258] T 583df4b8cc034a4eb335f0dccf4bf222 P d99c92f36fb3458582f7aaab81182991 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:20:46.096930 2798 ts_tablet_manager.cc:1905] T 583df4b8cc034a4eb335f0dccf4bf222 P 0dbe95cf940549da948f8adf2256841b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:20:46.097503 2796 ts_tablet_manager.cc:1905] T 583df4b8cc034a4eb335f0dccf4bf222 P d99c92f36fb3458582f7aaab81182991: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:20:46.098212 2797 ts_tablet_manager.cc:1905] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:20:46.099517 2798 ts_tablet_manager.cc:1918] T 583df4b8cc034a4eb335f0dccf4bf222 P 0dbe95cf940549da948f8adf2256841b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:20:46.099602 2798 log.cc:1196] T 583df4b8cc034a4eb335f0dccf4bf222 P 0dbe95cf940549da948f8adf2256841b: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-0/wal/wals/583df4b8cc034a4eb335f0dccf4bf222
I20260926 20:20:46.099877 2798 ts_tablet_manager.cc:1939] T 583df4b8cc034a4eb335f0dccf4bf222 P 0dbe95cf940549da948f8adf2256841b: Deleting consensus metadata
I20260926 20:20:46.100713 2198 catalog_manager.cc:4841] TS 0dbe95cf940549da948f8adf2256841b (127.0.116.193:35897): tablet 583df4b8cc034a4eb335f0dccf4bf222 (table test-table [id=45861cf3af72429a8d7c862347dc1917]) successfully deleted
I20260926 20:20:46.100857 2796 ts_tablet_manager.cc:1918] T 583df4b8cc034a4eb335f0dccf4bf222 P d99c92f36fb3458582f7aaab81182991: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:20:46.100946 2797 ts_tablet_manager.cc:1918] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:20:46.100986 2796 log.cc:1196] T 583df4b8cc034a4eb335f0dccf4bf222 P d99c92f36fb3458582f7aaab81182991: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/583df4b8cc034a4eb335f0dccf4bf222
I20260926 20:20:46.101018 2797 log.cc:1196] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/583df4b8cc034a4eb335f0dccf4bf222
I20260926 20:20:46.101272 2797 ts_tablet_manager.cc:1939] T 583df4b8cc034a4eb335f0dccf4bf222 P bd1532df667044eeb45599ac193e9617: Deleting consensus metadata
I20260926 20:20:46.101383 2796 ts_tablet_manager.cc:1939] T 583df4b8cc034a4eb335f0dccf4bf222 P d99c92f36fb3458582f7aaab81182991: Deleting consensus metadata
I20260926 20:20:46.101943 2200 catalog_manager.cc:4841] TS bd1532df667044eeb45599ac193e9617 (127.0.116.195:41685): tablet 583df4b8cc034a4eb335f0dccf4bf222 (table test-table [id=45861cf3af72429a8d7c862347dc1917]) successfully deleted
I20260926 20:20:46.102097 2201 catalog_manager.cc:4841] TS d99c92f36fb3458582f7aaab81182991 (127.0.116.194:38091): tablet 583df4b8cc034a4eb335f0dccf4bf222 (table test-table [id=45861cf3af72429a8d7c862347dc1917]) successfully deleted
Sep 26 20:20:46 dist-test-slave-qsm9 krb5kdc[2169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454046, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:20:46 dist-test-slave-qsm9 krb5kdc[2169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454046, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:46.126084 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:46.117573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39908 (local address 127.0.116.254:44587)
0926 20:20:46.117829 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:46.117835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:46.117851 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:20:46.117916 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:46.117919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:46.117962 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:46.118049 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:46.118745 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:46.119328 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:46.120250 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:46.120497 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:46.120583 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:46.123017 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:46.123021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:46.123023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:46.123047 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:46.124417 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:46.124838 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:46.124843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:46.124845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:46.124936 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:46.125206 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:46.125211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:46.125212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:46.125405 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:46.125544 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:46.125778 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 20:20:46.125906 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":103,"threads_started":1}
I20260926 20:20:46.128716 2212 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:20:46.129122 2212 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:20:46.133921 2491 tablet_service.cc:1467] Processing CreateTablet for tablet bab81e748dd045ee81c03ddd15bce953 (DEFAULT_TABLE table=test-table [id=e8638c75098140deb481b9dfb9df63f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:46.134044 2631 tablet_service.cc:1467] Processing CreateTablet for tablet bab81e748dd045ee81c03ddd15bce953 (DEFAULT_TABLE table=test-table [id=e8638c75098140deb481b9dfb9df63f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:46.134038 2351 tablet_service.cc:1467] Processing CreateTablet for tablet bab81e748dd045ee81c03ddd15bce953 (DEFAULT_TABLE table=test-table [id=e8638c75098140deb481b9dfb9df63f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:46.134248 2491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab81e748dd045ee81c03ddd15bce953. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:46.134306 2351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab81e748dd045ee81c03ddd15bce953. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:46.134328 2631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab81e748dd045ee81c03ddd15bce953. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:46.136698 2809 tablet_bootstrap.cc:492] T bab81e748dd045ee81c03ddd15bce953 P d99c92f36fb3458582f7aaab81182991: Bootstrap starting.
I20260926 20:20:46.136773 2807 tablet_bootstrap.cc:492] T bab81e748dd045ee81c03ddd15bce953 P bd1532df667044eeb45599ac193e9617: Bootstrap starting.
I20260926 20:20:46.137172 2808 tablet_bootstrap.cc:492] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b: Bootstrap starting.
I20260926 20:20:46.137588 2809 tablet_bootstrap.cc:654] T bab81e748dd045ee81c03ddd15bce953 P d99c92f36fb3458582f7aaab81182991: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:46.137645 2807 tablet_bootstrap.cc:654] T bab81e748dd045ee81c03ddd15bce953 P bd1532df667044eeb45599ac193e9617: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:46.138094 2808 tablet_bootstrap.cc:654] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:46.138739 2809 tablet_bootstrap.cc:492] T bab81e748dd045ee81c03ddd15bce953 P d99c92f36fb3458582f7aaab81182991: No bootstrap required, opened a new log
I20260926 20:20:46.138782 2807 tablet_bootstrap.cc:492] T bab81e748dd045ee81c03ddd15bce953 P bd1532df667044eeb45599ac193e9617: No bootstrap required, opened a new log
I20260926 20:20:46.138850 2807 ts_tablet_manager.cc:1397] T bab81e748dd045ee81c03ddd15bce953 P bd1532df667044eeb45599ac193e9617: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:20:46.138832 2809 ts_tablet_manager.cc:1397] T bab81e748dd045ee81c03ddd15bce953 P d99c92f36fb3458582f7aaab81182991: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:20:46.139181 2808 tablet_bootstrap.cc:492] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b: No bootstrap required, opened a new log
I20260926 20:20:46.139237 2808 ts_tablet_manager.cc:1397] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:20:46.139256 2809 raft_consensus.cc:348] T bab81e748dd045ee81c03ddd15bce953 P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:46.139256 2807 raft_consensus.cc:348] T bab81e748dd045ee81c03ddd15bce953 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:46.139361 2809 raft_consensus.cc:374] T bab81e748dd045ee81c03ddd15bce953 P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:46.139374 2807 raft_consensus.cc:374] T bab81e748dd045ee81c03ddd15bce953 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:46.139400 2809 raft_consensus.cc:729] T bab81e748dd045ee81c03ddd15bce953 P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d99c92f36fb3458582f7aaab81182991, State: Initialized, Role: FOLLOWER
I20260926 20:20:46.139408 2807 raft_consensus.cc:729] T bab81e748dd045ee81c03ddd15bce953 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1532df667044eeb45599ac193e9617, State: Initialized, Role: FOLLOWER
I20260926 20:20:46.139578 2808 raft_consensus.cc:348] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:46.139592 2809 consensus_queue.cc:260] T bab81e748dd045ee81c03ddd15bce953 P d99c92f36fb3458582f7aaab81182991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:46.139686 2808 raft_consensus.cc:374] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:46.139709 2808 raft_consensus.cc:729] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dbe95cf940549da948f8adf2256841b, State: Initialized, Role: FOLLOWER
I20260926 20:20:46.139832 2808 consensus_queue.cc:260] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:46.139991 2809 ts_tablet_manager.cc:1428] T bab81e748dd045ee81c03ddd15bce953 P d99c92f36fb3458582f7aaab81182991: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:20:46.140098 2808 ts_tablet_manager.cc:1428] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:20:46.140458 2807 consensus_queue.cc:260] T bab81e748dd045ee81c03ddd15bce953 P bd1532df667044eeb45599ac193e9617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:46.140766 2807 ts_tablet_manager.cc:1428] T bab81e748dd045ee81c03ddd15bce953 P bd1532df667044eeb45599ac193e9617: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:20:46.248459 2791 raft_consensus.cc:482] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:20:46.248595 2791 raft_consensus.cc:504] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:46.249060 2791 leader_election.cc:288] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d99c92f36fb3458582f7aaab81182991 (127.0.116.194:38091), bd1532df667044eeb45599ac193e9617 (127.0.116.195:41685)
I20260926 20:20:46.249569 2637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab81e748dd045ee81c03ddd15bce953" candidate_uuid: "0dbe95cf940549da948f8adf2256841b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd1532df667044eeb45599ac193e9617" is_pre_election: true
I20260926 20:20:46.249570 2510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab81e748dd045ee81c03ddd15bce953" candidate_uuid: "0dbe95cf940549da948f8adf2256841b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99c92f36fb3458582f7aaab81182991" is_pre_election: true
I20260926 20:20:46.249733 2510 raft_consensus.cc:2454] T bab81e748dd045ee81c03ddd15bce953 P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dbe95cf940549da948f8adf2256841b in term 0.
I20260926 20:20:46.249733 2637 raft_consensus.cc:2454] T bab81e748dd045ee81c03ddd15bce953 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dbe95cf940549da948f8adf2256841b in term 0.
I20260926 20:20:46.250111 2305 leader_election.cc:302] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0dbe95cf940549da948f8adf2256841b, bd1532df667044eeb45599ac193e9617, d99c92f36fb3458582f7aaab81182991; no voters:
I20260926 20:20:46.250278 2791 raft_consensus.cc:2781] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:20:46.250339 2791 raft_consensus.cc:482] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:20:46.250393 2791 raft_consensus.cc:3037] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:46.251379 2791 raft_consensus.cc:504] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:46.251748 2791 leader_election.cc:288] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b [CANDIDATE]: Term 1 election: Requested vote from peers d99c92f36fb3458582f7aaab81182991 (127.0.116.194:38091), bd1532df667044eeb45599ac193e9617 (127.0.116.195:41685)
I20260926 20:20:46.252063 2510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab81e748dd045ee81c03ddd15bce953" candidate_uuid: "0dbe95cf940549da948f8adf2256841b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99c92f36fb3458582f7aaab81182991"
I20260926 20:20:46.252137 2637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab81e748dd045ee81c03ddd15bce953" candidate_uuid: "0dbe95cf940549da948f8adf2256841b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd1532df667044eeb45599ac193e9617"
I20260926 20:20:46.252214 2510 raft_consensus.cc:3037] T bab81e748dd045ee81c03ddd15bce953 P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:46.252278 2637 raft_consensus.cc:3037] T bab81e748dd045ee81c03ddd15bce953 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:46.253142 2510 raft_consensus.cc:2454] T bab81e748dd045ee81c03ddd15bce953 P d99c92f36fb3458582f7aaab81182991 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dbe95cf940549da948f8adf2256841b in term 1.
I20260926 20:20:46.253226 2637 raft_consensus.cc:2454] T bab81e748dd045ee81c03ddd15bce953 P bd1532df667044eeb45599ac193e9617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dbe95cf940549da948f8adf2256841b in term 1.
I20260926 20:20:46.253465 2305 leader_election.cc:302] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dbe95cf940549da948f8adf2256841b, d99c92f36fb3458582f7aaab81182991; no voters:
I20260926 20:20:46.253621 2791 raft_consensus.cc:2781] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:46.253703 2791 raft_consensus.cc:686] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b [term 1 LEADER]: Becoming Leader. State: Replica: 0dbe95cf940549da948f8adf2256841b, State: Running, Role: LEADER
I20260926 20:20:46.253870 2791 consensus_queue.cc:237] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:46.255647 2211 catalog_manager.cc:5495] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dbe95cf940549da948f8adf2256841b (127.0.116.193). New cstate: current_term: 1 leader_uuid: "0dbe95cf940549da948f8adf2256841b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } health_report { overall_health: UNKNOWN } } }
I20260926 20:20:46.270184 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:46.267284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48700 (local address 127.0.116.193:35897)
0926 20:20:46.267585 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:46.267592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:46.267603 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:20:46.267638 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:46.267640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:46.267678 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:46.267746 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:20:46.267964 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:46.268468 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:46.269241 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:46.269463 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:46.269537 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:46.269729 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:20:46.269793 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:20:46.269844 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:46.269940 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 20:20:46.269994 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":117,"threads_started":1}
I20260926 20:20:46.278493 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:46.275685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:53273 (local address 127.0.116.193:35897)
0926 20:20:46.275809 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:46.275812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:46.275939 (+ 127us) server_negotiation.cc:407] Connection header received
0926 20:20:46.276036 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:46.276038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:46.276075 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:46.276134 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:20:46.276327 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:46.276805 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:46.277570 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:46.277766 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:46.277834 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:46.278085 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:20:46.278169 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:20:46.278236 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:46.278326 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 20:20:46.278371 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:20:46.278508 2819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:46.275573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:35897 (local address 127.0.116.193:53273)
0926 20:20:46.275850 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:46.275862 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:46.275960 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:46.276155 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:46.276157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:46.276163 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:20:46.276248 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:46.276251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:46.276899 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:46.276901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:46.277479 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:46.277487 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:46.277861 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:46.277863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:46.277935 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:46.277960 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:20:46.278254 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:20:46.278260 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:20:46.278310 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 20:20:46.278376 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":80,"threads_started":1}
I20260926 20:20:46.284394 2637 raft_consensus.cc:1261] T bab81e748dd045ee81c03ddd15bce953 P bd1532df667044eeb45599ac193e9617 [term 1 FOLLOWER]: Refusing update from remote peer 0dbe95cf940549da948f8adf2256841b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:46.284447 2510 raft_consensus.cc:1261] T bab81e748dd045ee81c03ddd15bce953 P d99c92f36fb3458582f7aaab81182991 [term 1 FOLLOWER]: Refusing update from remote peer 0dbe95cf940549da948f8adf2256841b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:46.284780 2727 consensus_queue.cc:1035] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:20:46.284976 2791 consensus_queue.cc:1035] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:20:46.317677 2211 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:20:46.317855 2211 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:20:46.319960 2211 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972: Sending DeleteTablet for 3 replicas of tablet bab81e748dd045ee81c03ddd15bce953
I20260926 20:20:46.320468 2350 tablet_service.cc:1514] Processing DeleteTablet for tablet bab81e748dd045ee81c03ddd15bce953 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:46 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:48690
I20260926 20:20:46.320664 2798 tablet_replica.cc:330] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b: stopping tablet replica
I20260926 20:20:46.320776 2798 raft_consensus.cc:2229] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:20:46.320749 2631 tablet_service.cc:1514] Processing DeleteTablet for tablet bab81e748dd045ee81c03ddd15bce953 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:46 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:51448
I20260926 20:20:46.320940 2797 tablet_replica.cc:330] T bab81e748dd045ee81c03ddd15bce953 P bd1532df667044eeb45599ac193e9617: stopping tablet replica
I20260926 20:20:46.321053 2797 raft_consensus.cc:2229] T bab81e748dd045ee81c03ddd15bce953 P bd1532df667044eeb45599ac193e9617 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:20:46.321055 2798 raft_consensus.cc:2258] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:20:46.321146 2797 raft_consensus.cc:2258] T bab81e748dd045ee81c03ddd15bce953 P bd1532df667044eeb45599ac193e9617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:20:46.321262 2491 tablet_service.cc:1514] Processing DeleteTablet for tablet bab81e748dd045ee81c03ddd15bce953 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:46 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:40586
I20260926 20:20:46.321482 2796 tablet_replica.cc:330] T bab81e748dd045ee81c03ddd15bce953 P d99c92f36fb3458582f7aaab81182991: stopping tablet replica
I20260926 20:20:46.321585 2796 raft_consensus.cc:2229] T bab81e748dd045ee81c03ddd15bce953 P d99c92f36fb3458582f7aaab81182991 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:20:46.321683 2796 raft_consensus.cc:2258] T bab81e748dd045ee81c03ddd15bce953 P d99c92f36fb3458582f7aaab81182991 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:20:46.321971 2797 ts_tablet_manager.cc:1905] T bab81e748dd045ee81c03ddd15bce953 P bd1532df667044eeb45599ac193e9617: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:20:46.322631 2796 ts_tablet_manager.cc:1905] T bab81e748dd045ee81c03ddd15bce953 P d99c92f36fb3458582f7aaab81182991: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:20:46.323961 2798 ts_tablet_manager.cc:1905] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:20:46.324497 2797 ts_tablet_manager.cc:1918] T bab81e748dd045ee81c03ddd15bce953 P bd1532df667044eeb45599ac193e9617: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:20:46.324560 2797 log.cc:1196] T bab81e748dd045ee81c03ddd15bce953 P bd1532df667044eeb45599ac193e9617: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/bab81e748dd045ee81c03ddd15bce953
I20260926 20:20:46.324795 2797 ts_tablet_manager.cc:1939] T bab81e748dd045ee81c03ddd15bce953 P bd1532df667044eeb45599ac193e9617: Deleting consensus metadata
I20260926 20:20:46.325086 2796 ts_tablet_manager.cc:1918] T bab81e748dd045ee81c03ddd15bce953 P d99c92f36fb3458582f7aaab81182991: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:20:46.325150 2796 log.cc:1196] T bab81e748dd045ee81c03ddd15bce953 P d99c92f36fb3458582f7aaab81182991: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/bab81e748dd045ee81c03ddd15bce953
I20260926 20:20:46.325403 2796 ts_tablet_manager.cc:1939] T bab81e748dd045ee81c03ddd15bce953 P d99c92f36fb3458582f7aaab81182991: Deleting consensus metadata
I20260926 20:20:46.325519 2200 catalog_manager.cc:4841] TS bd1532df667044eeb45599ac193e9617 (127.0.116.195:41685): tablet bab81e748dd045ee81c03ddd15bce953 (table test-table [id=e8638c75098140deb481b9dfb9df63f4]) successfully deleted
I20260926 20:20:46.326165 2201 catalog_manager.cc:4841] TS d99c92f36fb3458582f7aaab81182991 (127.0.116.194:38091): tablet bab81e748dd045ee81c03ddd15bce953 (table test-table [id=e8638c75098140deb481b9dfb9df63f4]) successfully deleted
I20260926 20:20:46.326483 2798 ts_tablet_manager.cc:1918] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:20:46.326536 2798 log.cc:1196] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-0/wal/wals/bab81e748dd045ee81c03ddd15bce953
I20260926 20:20:46.326709 2798 ts_tablet_manager.cc:1939] T bab81e748dd045ee81c03ddd15bce953 P 0dbe95cf940549da948f8adf2256841b: Deleting consensus metadata
I20260926 20:20:46.327395 2198 catalog_manager.cc:4841] TS 0dbe95cf940549da948f8adf2256841b (127.0.116.193:35897): tablet bab81e748dd045ee81c03ddd15bce953 (table test-table [id=e8638c75098140deb481b9dfb9df63f4]) successfully deleted
I20260926 20:20:46.327705 2211 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:20:46.328099 2211 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:20:46.332372 2350 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7743fcdf664983b38ff5ba259828b3 (DEFAULT_TABLE table=test-table [id=5b326be06c2c41f790652baec4e68b0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:46.332505 2631 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7743fcdf664983b38ff5ba259828b3 (DEFAULT_TABLE table=test-table [id=5b326be06c2c41f790652baec4e68b0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:46.332561 2491 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7743fcdf664983b38ff5ba259828b3 (DEFAULT_TABLE table=test-table [id=5b326be06c2c41f790652baec4e68b0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:46.332639 2350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7743fcdf664983b38ff5ba259828b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:46.332717 2631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7743fcdf664983b38ff5ba259828b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:46.332787 2491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7743fcdf664983b38ff5ba259828b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:46.334875 2807 tablet_bootstrap.cc:492] T 2d7743fcdf664983b38ff5ba259828b3 P bd1532df667044eeb45599ac193e9617: Bootstrap starting.
I20260926 20:20:46.335038 2809 tablet_bootstrap.cc:492] T 2d7743fcdf664983b38ff5ba259828b3 P d99c92f36fb3458582f7aaab81182991: Bootstrap starting.
I20260926 20:20:46.335618 2807 tablet_bootstrap.cc:654] T 2d7743fcdf664983b38ff5ba259828b3 P bd1532df667044eeb45599ac193e9617: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:46.335958 2809 tablet_bootstrap.cc:654] T 2d7743fcdf664983b38ff5ba259828b3 P d99c92f36fb3458582f7aaab81182991: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:46.336827 2807 tablet_bootstrap.cc:492] T 2d7743fcdf664983b38ff5ba259828b3 P bd1532df667044eeb45599ac193e9617: No bootstrap required, opened a new log
I20260926 20:20:46.336897 2807 ts_tablet_manager.cc:1397] T 2d7743fcdf664983b38ff5ba259828b3 P bd1532df667044eeb45599ac193e9617: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 20:20:46.337240 2807 raft_consensus.cc:348] T 2d7743fcdf664983b38ff5ba259828b3 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:46.337330 2807 raft_consensus.cc:374] T 2d7743fcdf664983b38ff5ba259828b3 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:46.337352 2807 raft_consensus.cc:729] T 2d7743fcdf664983b38ff5ba259828b3 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1532df667044eeb45599ac193e9617, State: Initialized, Role: FOLLOWER
I20260926 20:20:46.337500 2807 consensus_queue.cc:260] T 2d7743fcdf664983b38ff5ba259828b3 P bd1532df667044eeb45599ac193e9617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:46.337771 2807 ts_tablet_manager.cc:1428] T 2d7743fcdf664983b38ff5ba259828b3 P bd1532df667044eeb45599ac193e9617: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 20:20:46.338675 2808 tablet_bootstrap.cc:492] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b: Bootstrap starting.
I20260926 20:20:46.339473 2808 tablet_bootstrap.cc:654] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:46.339622 2809 tablet_bootstrap.cc:492] T 2d7743fcdf664983b38ff5ba259828b3 P d99c92f36fb3458582f7aaab81182991: No bootstrap required, opened a new log
I20260926 20:20:46.339707 2809 ts_tablet_manager.cc:1397] T 2d7743fcdf664983b38ff5ba259828b3 P d99c92f36fb3458582f7aaab81182991: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260926 20:20:46.340167 2809 raft_consensus.cc:348] T 2d7743fcdf664983b38ff5ba259828b3 P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:46.340286 2809 raft_consensus.cc:374] T 2d7743fcdf664983b38ff5ba259828b3 P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:46.340317 2809 raft_consensus.cc:729] T 2d7743fcdf664983b38ff5ba259828b3 P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d99c92f36fb3458582f7aaab81182991, State: Initialized, Role: FOLLOWER
I20260926 20:20:46.340461 2809 consensus_queue.cc:260] T 2d7743fcdf664983b38ff5ba259828b3 P d99c92f36fb3458582f7aaab81182991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:46.340724 2809 ts_tablet_manager.cc:1428] T 2d7743fcdf664983b38ff5ba259828b3 P d99c92f36fb3458582f7aaab81182991: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:20:46.340987 2808 tablet_bootstrap.cc:492] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b: No bootstrap required, opened a new log
I20260926 20:20:46.341058 2808 ts_tablet_manager.cc:1397] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260926 20:20:46.341426 2808 raft_consensus.cc:348] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:46.341521 2808 raft_consensus.cc:374] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:46.341550 2808 raft_consensus.cc:729] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dbe95cf940549da948f8adf2256841b, State: Initialized, Role: FOLLOWER
I20260926 20:20:46.341661 2808 consensus_queue.cc:260] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:46.341889 2808 ts_tablet_manager.cc:1428] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:20:46.437096 2791 raft_consensus.cc:482] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:20:46.437218 2791 raft_consensus.cc:504] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:46.437673 2791 leader_election.cc:288] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d99c92f36fb3458582f7aaab81182991 (127.0.116.194:38091), bd1532df667044eeb45599ac193e9617 (127.0.116.195:41685)
I20260926 20:20:46.438098 2637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7743fcdf664983b38ff5ba259828b3" candidate_uuid: "0dbe95cf940549da948f8adf2256841b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd1532df667044eeb45599ac193e9617" is_pre_election: true
I20260926 20:20:46.438102 2510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7743fcdf664983b38ff5ba259828b3" candidate_uuid: "0dbe95cf940549da948f8adf2256841b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99c92f36fb3458582f7aaab81182991" is_pre_election: true
I20260926 20:20:46.438253 2637 raft_consensus.cc:2454] T 2d7743fcdf664983b38ff5ba259828b3 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dbe95cf940549da948f8adf2256841b in term 0.
I20260926 20:20:46.438268 2510 raft_consensus.cc:2454] T 2d7743fcdf664983b38ff5ba259828b3 P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dbe95cf940549da948f8adf2256841b in term 0.
I20260926 20:20:46.438583 2304 leader_election.cc:302] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dbe95cf940549da948f8adf2256841b, bd1532df667044eeb45599ac193e9617; no voters:
I20260926 20:20:46.438757 2791 raft_consensus.cc:2781] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:20:46.438817 2791 raft_consensus.cc:482] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:20:46.438861 2791 raft_consensus.cc:3037] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:46.439982 2791 raft_consensus.cc:504] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:46.440423 2791 leader_election.cc:288] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b [CANDIDATE]: Term 1 election: Requested vote from peers d99c92f36fb3458582f7aaab81182991 (127.0.116.194:38091), bd1532df667044eeb45599ac193e9617 (127.0.116.195:41685)
I20260926 20:20:46.440706 2510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7743fcdf664983b38ff5ba259828b3" candidate_uuid: "0dbe95cf940549da948f8adf2256841b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99c92f36fb3458582f7aaab81182991"
I20260926 20:20:46.440850 2510 raft_consensus.cc:3037] T 2d7743fcdf664983b38ff5ba259828b3 P d99c92f36fb3458582f7aaab81182991 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:46.440838 2637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7743fcdf664983b38ff5ba259828b3" candidate_uuid: "0dbe95cf940549da948f8adf2256841b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd1532df667044eeb45599ac193e9617"
I20260926 20:20:46.440977 2637 raft_consensus.cc:3037] T 2d7743fcdf664983b38ff5ba259828b3 P bd1532df667044eeb45599ac193e9617 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:46.442070 2510 raft_consensus.cc:2454] T 2d7743fcdf664983b38ff5ba259828b3 P d99c92f36fb3458582f7aaab81182991 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dbe95cf940549da948f8adf2256841b in term 1.
I20260926 20:20:46.442072 2637 raft_consensus.cc:2454] T 2d7743fcdf664983b38ff5ba259828b3 P bd1532df667044eeb45599ac193e9617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dbe95cf940549da948f8adf2256841b in term 1.
I20260926 20:20:46.442492 2304 leader_election.cc:302] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dbe95cf940549da948f8adf2256841b, bd1532df667044eeb45599ac193e9617; no voters:
I20260926 20:20:46.442715 2791 raft_consensus.cc:2781] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:46.442809 2791 raft_consensus.cc:686] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b [term 1 LEADER]: Becoming Leader. State: Replica: 0dbe95cf940549da948f8adf2256841b, State: Running, Role: LEADER
I20260926 20:20:46.442976 2791 consensus_queue.cc:237] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } }
I20260926 20:20:46.444694 2212 catalog_manager.cc:5495] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dbe95cf940549da948f8adf2256841b (127.0.116.193). New cstate: current_term: 1 leader_uuid: "0dbe95cf940549da948f8adf2256841b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbe95cf940549da948f8adf2256841b" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 } health_report { overall_health: UNKNOWN } } }
I20260926 20:20:46.474555 2637 raft_consensus.cc:1261] T 2d7743fcdf664983b38ff5ba259828b3 P bd1532df667044eeb45599ac193e9617 [term 1 FOLLOWER]: Refusing update from remote peer 0dbe95cf940549da948f8adf2256841b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:46.474558 2510 raft_consensus.cc:1261] T 2d7743fcdf664983b38ff5ba259828b3 P d99c92f36fb3458582f7aaab81182991 [term 1 FOLLOWER]: Refusing update from remote peer 0dbe95cf940549da948f8adf2256841b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:46.474983 2791 consensus_queue.cc:1035] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99c92f36fb3458582f7aaab81182991" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:20:46.475133 2822 consensus_queue.cc:1035] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd1532df667044eeb45599ac193e9617" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 41685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 20:20:46.487641 2776 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 20:20:48.038129 2212 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:20:48.038312 2212 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:20:48.040804 2212 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4137e3e36eb477b87fee13fafe0d972: Sending DeleteTablet for 3 replicas of tablet 2d7743fcdf664983b38ff5ba259828b3
I20260926 20:20:48.041415 2350 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7743fcdf664983b38ff5ba259828b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:48 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:48690
I20260926 20:20:48.041492 2491 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7743fcdf664983b38ff5ba259828b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:48 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:40586
I20260926 20:20:48.041734 2631 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7743fcdf664983b38ff5ba259828b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:48 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:51448
I20260926 20:20:48.041779 2849 tablet_replica.cc:330] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b: stopping tablet replica
I20260926 20:20:48.041940 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 2286
I20260926 20:20:48.041955 2849 raft_consensus.cc:2229] T 2d7743fcdf664983b38ff5ba259828b3 P 0dbe95cf940549da948f8adf2256841b [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:20:48.042819 2851 tablet_replica.cc:330] T 2d7743fcdf664983b38ff5ba259828b3 P bd1532df667044eeb45599ac193e9617: stopping tablet replica
I20260926 20:20:48.043013 2851 raft_consensus.cc:2229] T 2d7743fcdf664983b38ff5ba259828b3 P bd1532df667044eeb45599ac193e9617 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:20:48.043145 2851 raft_consensus.cc:2258] T 2d7743fcdf664983b38ff5ba259828b3 P bd1532df667044eeb45599ac193e9617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:20:48.043893 2851 ts_tablet_manager.cc:1905] T 2d7743fcdf664983b38ff5ba259828b3 P bd1532df667044eeb45599ac193e9617: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:20:48.047444 2851 ts_tablet_manager.cc:1918] T 2d7743fcdf664983b38ff5ba259828b3 P bd1532df667044eeb45599ac193e9617: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:20:48.047564 2851 log.cc:1196] T 2d7743fcdf664983b38ff5ba259828b3 P bd1532df667044eeb45599ac193e9617: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/2d7743fcdf664983b38ff5ba259828b3
I20260926 20:20:48.047603 2850 tablet_replica.cc:330] T 2d7743fcdf664983b38ff5ba259828b3 P d99c92f36fb3458582f7aaab81182991: stopping tablet replica
I20260926 20:20:48.047794 2850 raft_consensus.cc:2229] T 2d7743fcdf664983b38ff5ba259828b3 P d99c92f36fb3458582f7aaab81182991 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:20:48.047900 2851 ts_tablet_manager.cc:1939] T 2d7743fcdf664983b38ff5ba259828b3 P bd1532df667044eeb45599ac193e9617: Deleting consensus metadata
I20260926 20:20:48.047956 2850 raft_consensus.cc:2258] T 2d7743fcdf664983b38ff5ba259828b3 P d99c92f36fb3458582f7aaab81182991 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:20:48.048646 2850 ts_tablet_manager.cc:1905] T 2d7743fcdf664983b38ff5ba259828b3 P d99c92f36fb3458582f7aaab81182991: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:20:48.048758 2200 catalog_manager.cc:4841] TS bd1532df667044eeb45599ac193e9617 (127.0.116.195:41685): tablet 2d7743fcdf664983b38ff5ba259828b3 (table test-table [id=5b326be06c2c41f790652baec4e68b0d]) successfully deleted
W20260926 20:20:48.050848 2198 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.116.193:35897 (error 108)
I20260926 20:20:48.051192 2850 ts_tablet_manager.cc:1918] T 2d7743fcdf664983b38ff5ba259828b3 P d99c92f36fb3458582f7aaab81182991: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:20:48.051275 2850 log.cc:1196] T 2d7743fcdf664983b38ff5ba259828b3 P d99c92f36fb3458582f7aaab81182991: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/2d7743fcdf664983b38ff5ba259828b3
I20260926 20:20:48.051570 2850 ts_tablet_manager.cc:1939] T 2d7743fcdf664983b38ff5ba259828b3 P d99c92f36fb3458582f7aaab81182991: Deleting consensus metadata
I20260926 20:20:48.051786 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 2426
I20260926 20:20:48.052590 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:48.051687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:35897 (local address 127.0.0.1:48710)
0926 20:20:48.052433 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:48.052503 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.193:35897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":86,"threads_started":1}
W20260926 20:20:48.052835 2198 catalog_manager.cc:4584] TS 0dbe95cf940549da948f8adf2256841b (127.0.116.193:35897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d7743fcdf664983b38ff5ba259828b3: Network error: Client connection negotiation failed: client connection to 127.0.116.193:35897: connect: Connection refused (error 111)
I20260926 20:20:48.058637 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 2566
I20260926 20:20:48.059520 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:48.058822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:38091 (local address 127.0.0.1:40590)
0926 20:20:48.059322 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:48.059396 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.194:38091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398}
I20260926 20:20:48.065303 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 2182
2026-09-26T20:20:48Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6217 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:48 dist-test-slave-qsm9 krb5kdc[2856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:48 dist-test-slave-qsm9 krb5kdc[2856](info): set up 2 sockets
Sep 26 20:20:48 dist-test-slave-qsm9 krb5kdc[2856](info): commencing operation
krb5kdc: starting...
W20260926 20:20:49.152081 467 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:20:49 dist-test-slave-qsm9 krb5kdc[2856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:20:49.284298 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:38399
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:38399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:49.400748 2869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:49.401082 2869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:49.401170 2869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:49.405864 2869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:49.405937 2869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:49.405957 2869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:49.405974 2869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:49.405990 2869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:49.411532 2869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:38399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:38399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:49.412621 2869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:49.413617 2869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:49.422048 2874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:49.422060 2877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:49.422060 2875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:49.422482 2869 server_base.cc:1034] running on GCE node
I20260926 20:20:49.423070 2869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:49.424219 2869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:49.425412 2869 hybrid_clock.cc:648] HybridClock initialized: now 1790454049425312 us; error 116 us; skew 500 ppm
Sep 26 20:20:49 dist-test-slave-qsm9 krb5kdc[2856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:49.428198 2869 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:20:49.429054 2869 webserver.cc:460] Webserver started at http://127.0.116.254:40059/ using document root <none> and password file <none>
I20260926 20:20:49.429605 2869 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:49.429690 2869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:49.429926 2869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:49.431995 2869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "ebdfacc591ef4193a042393637c60ed2"
format_stamp: "Formatted at 2026-09-26 20:20:49 on dist-test-slave-qsm9"
server_key: "544854f9af3f6f8dc376ce5cbcd455d3"
server_key_iv: "793fca3a465b41b600e72da2e3086844"
server_key_version: "encryptionkey@0"
I20260926 20:20:49.432551 2869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "ebdfacc591ef4193a042393637c60ed2"
format_stamp: "Formatted at 2026-09-26 20:20:49 on dist-test-slave-qsm9"
server_key: "544854f9af3f6f8dc376ce5cbcd455d3"
server_key_iv: "793fca3a465b41b600e72da2e3086844"
server_key_version: "encryptionkey@0"
I20260926 20:20:49.436371 2869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:20:49.439103 2884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:49.440687 2869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:20:49.440840 2869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "ebdfacc591ef4193a042393637c60ed2"
format_stamp: "Formatted at 2026-09-26 20:20:49 on dist-test-slave-qsm9"
server_key: "544854f9af3f6f8dc376ce5cbcd455d3"
server_key_iv: "793fca3a465b41b600e72da2e3086844"
server_key_version: "encryptionkey@0"
I20260926 20:20:49.440979 2869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:49.460225 2869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:49.463375 2869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:49.463618 2869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:49.470582 2869 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:38399
I20260926 20:20:49.470638 2936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:38399 every 8 connection(s)
I20260926 20:20:49.471658 2869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:49.474499 2937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:49.480077 2937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2: Bootstrap starting.
I20260926 20:20:49.481047 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 2869
I20260926 20:20:49.481186 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:20:49.481462 467 external_mini_cluster.cc:1428] Setting key 7e627ed3851545a7e95ce47696fe7ff9
I20260926 20:20:49.482887 2937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:49.483862 2937 log.cc:824] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:49.486048 2937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2: No bootstrap required, opened a new log
I20260926 20:20:49.490159 2937 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdfacc591ef4193a042393637c60ed2" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 38399 } }
I20260926 20:20:49.490466 2937 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:49.490515 2937 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebdfacc591ef4193a042393637c60ed2, State: Initialized, Role: FOLLOWER
Sep 26 20:20:49 dist-test-slave-qsm9 krb5kdc[2856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:49.491011 2937 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdfacc591ef4193a042393637c60ed2" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 38399 } }
I20260926 20:20:49.491171 2937 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:49.491259 2937 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:49.491364 2937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:49.492321 2937 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdfacc591ef4193a042393637c60ed2" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 38399 } }
I20260926 20:20:49.492694 2937 leader_election.cc:302] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebdfacc591ef4193a042393637c60ed2; no voters:
I20260926 20:20:49.493076 2937 leader_election.cc:288] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:49.493187 2942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:49.493399 2942 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2 [term 1 LEADER]: Becoming Leader. State: Replica: ebdfacc591ef4193a042393637c60ed2, State: Running, Role: LEADER
I20260926 20:20:49.493810 2942 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdfacc591ef4193a042393637c60ed2" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 38399 } }
I20260926 20:20:49.494434 2937 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:49.495571 2943 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebdfacc591ef4193a042393637c60ed2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdfacc591ef4193a042393637c60ed2" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 38399 } } }
I20260926 20:20:49.495698 2943 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:49.495914 2943 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebdfacc591ef4193a042393637c60ed2. Latest consensus state: current_term: 1 leader_uuid: "ebdfacc591ef4193a042393637c60ed2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdfacc591ef4193a042393637c60ed2" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 38399 } } }
I20260926 20:20:49.495991 2943 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:49.497656 2940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:49.482636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32808 (local address 127.0.116.254:38399)
0926 20:20:49.483149 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:49.483172 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:49.483221 (+ 49us) server_negotiation.cc:407] Connection header received
0926 20:20:49.484080 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:49.484104 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:49.484488 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:49.484877 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:49.485521 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:49.487011 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:49.487930 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:49.488665 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:49.488851 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:49.491603 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:49.491608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:49.491621 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:49.491647 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:49.493702 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:49.494201 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:49.494207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:49.494213 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:49.494353 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:49.494662 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:49.494665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:49.494667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:49.495236 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:49.496232 (+ 996us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:49.496398 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 20:20:49.496726 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":144,"threads_started":1}
W20260926 20:20:49.500041 2957 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:20:49.500134 2957 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:20:49.500195 2951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:49.500885 2951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:49.506378 2951 catalog_manager.cc:1347] Generated new cluster ID: f949fa8ce07e43d39676a95b0bf51f11
I20260926 20:20:49.506466 2951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:49.517585 2951 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:49.518983 2951 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:49.523653 2951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebdfacc591ef4193a042393637c60ed2: Generated new TSK 0
I20260926 20:20:49.524359 2951 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
I20260926 20:20:49.559464 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:38399
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:49.692895 2963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:49.693192 2963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:49.693279 2963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:49.697657 2963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:49.697719 2963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:49.697811 2963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:20:49.703816 2963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:38399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:49.704968 2963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:49.706017 2963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:49.715773 2969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:49.715773 2968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:49.715803 2971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:49.716307 2963 server_base.cc:1034] running on GCE node
I20260926 20:20:49.716750 2963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:49.717387 2963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:49.718564 2963 hybrid_clock.cc:648] HybridClock initialized: now 1790454049718521 us; error 64 us; skew 500 ppm
Sep 26 20:20:49 dist-test-slave-qsm9 krb5kdc[2856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:49.721561 2963 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:20:49.722517 2963 webserver.cc:460] Webserver started at http://127.0.116.193:42893/ using document root <none> and password file <none>
I20260926 20:20:49.723107 2963 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:49.723203 2963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:49.723455 2963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:49.725653 2963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "d54b7da467c047e29f78211ecd0516c4"
format_stamp: "Formatted at 2026-09-26 20:20:49 on dist-test-slave-qsm9"
server_key: "5bd2f2722d7596fbe424c061c8803c71"
server_key_iv: "d6bfd0b3f89c43fdf6cb126b5e4f254b"
server_key_version: "encryptionkey@0"
I20260926 20:20:49.726280 2963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "d54b7da467c047e29f78211ecd0516c4"
format_stamp: "Formatted at 2026-09-26 20:20:49 on dist-test-slave-qsm9"
server_key: "5bd2f2722d7596fbe424c061c8803c71"
server_key_iv: "d6bfd0b3f89c43fdf6cb126b5e4f254b"
server_key_version: "encryptionkey@0"
I20260926 20:20:49.730228 2963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 20:20:49.732863 2978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:49.734160 2963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:20:49.734332 2963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "d54b7da467c047e29f78211ecd0516c4"
format_stamp: "Formatted at 2026-09-26 20:20:49 on dist-test-slave-qsm9"
server_key: "5bd2f2722d7596fbe424c061c8803c71"
server_key_iv: "d6bfd0b3f89c43fdf6cb126b5e4f254b"
server_key_version: "encryptionkey@0"
I20260926 20:20:49.734455 2963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:49.765038 2963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:49.768584 2963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:49.768803 2963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:49.769431 2963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:20:49.770298 2963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:49.770368 2963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:49.770442 2963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:49.770485 2963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:49.779804 2963 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:43933
I20260926 20:20:49.779824 3091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:43933 every 8 connection(s)
I20260926 20:20:49.780730 2963 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:20:49.787866 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 2963
I20260926 20:20:49.788020 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:20:49.788336 467 external_mini_cluster.cc:1428] Setting key 71f8d858075fbcd1ce0eea4be2aa165b
Sep 26 20:20:49 dist-test-slave-qsm9 krb5kdc[2856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:49.796025 2940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:49.782656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:44321 (local address 127.0.116.254:38399)
0926 20:20:49.782814 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:49.782818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:49.783857 (+ 1039us) server_negotiation.cc:407] Connection header received
0926 20:20:49.785235 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:49.785238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:49.785287 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:49.785369 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:49.786673 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:49.787207 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:49.788751 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:49.789069 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:49.789198 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:49.791997 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:49.792000 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:49.792002 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:49.792026 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:49.793618 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:49.794259 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:49.794265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:49.794268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:49.794358 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:49.794707 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:49.794711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:49.794712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:49.794951 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:49.795064 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:49.795713 (+ 649us) server_negotiation.cc:299] Negotiation successful
0926 20:20:49.795839 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:20:49.796904 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:49.783118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:38399 (local address 127.0.116.193:44321)
0926 20:20:49.783650 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:49.783696 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:49.785004 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:49.785573 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:49.785581 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:49.785971 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:49.786497 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:49.786513 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:49.787370 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:49.787373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:49.788607 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:49.788617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:49.789246 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:49.789251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:49.789440 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:49.790261 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:49.790279 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:49.791815 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:49.793774 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:49.793780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:49.793797 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:49.794150 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:49.794463 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:49.794466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:49.794468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:49.794608 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:49.795061 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:49.795068 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:49.795415 (+ 347us) client_negotiation.cc:769] Sending connection context
0926 20:20:49.795660 (+ 245us) client_negotiation.cc:240] Negotiation successful
0926 20:20:49.795992 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.0.116.194@KRBTEST.COM; defaulting to no policy
I20260926 20:20:49.798132 3092 heartbeater.cc:347] Connected to a master server at 127.0.116.254:38399
I20260926 20:20:49.798379 3092 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:49.798866 3092 heartbeater.cc:511] Master 127.0.116.254:38399 requested a full tablet report, sending...
I20260926 20:20:49.800419 2899 ts_manager.cc:194] Registered new tserver with Master: d54b7da467c047e29f78211ecd0516c4 (127.0.116.193:43933)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.194@KRBTEST.COM" created.
I20260926 20:20:49.801860 2899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:44321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
I20260926 20:20:49.819659 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:38399
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:49.931011 3097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:49.931280 3097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:49.931352 3097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:49.935467 3097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:49.935622 3097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:49.935745 3097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:20:49.941244 3097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:38399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:49.942323 3097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:49.943569 3097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:49.953285 3102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:49.953369 3103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:49.953285 3105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:49.953500 3097 server_base.cc:1034] running on GCE node
I20260926 20:20:49.953974 3097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:49.954612 3097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:49.955816 3097 hybrid_clock.cc:648] HybridClock initialized: now 1790454049955782 us; error 47 us; skew 500 ppm
Sep 26 20:20:49 dist-test-slave-qsm9 krb5kdc[2856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:49.958514 3097 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:20:49.959336 3097 webserver.cc:460] Webserver started at http://127.0.116.194:34715/ using document root <none> and password file <none>
I20260926 20:20:49.959949 3097 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:49.960011 3097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:49.960217 3097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:49.962236 3097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "5f80438615734ee2a5855ac807283a5b"
format_stamp: "Formatted at 2026-09-26 20:20:49 on dist-test-slave-qsm9"
server_key: "97310bd2ab48c834b38e0869193134c3"
server_key_iv: "02a9e1a3e3657cdb6d1442d760b523e5"
server_key_version: "encryptionkey@0"
I20260926 20:20:49.962828 3097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "5f80438615734ee2a5855ac807283a5b"
format_stamp: "Formatted at 2026-09-26 20:20:49 on dist-test-slave-qsm9"
server_key: "97310bd2ab48c834b38e0869193134c3"
server_key_iv: "02a9e1a3e3657cdb6d1442d760b523e5"
server_key_version: "encryptionkey@0"
I20260926 20:20:49.966917 3097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:20:49.969604 3112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:49.970983 3097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:20:49.971134 3097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "5f80438615734ee2a5855ac807283a5b"
format_stamp: "Formatted at 2026-09-26 20:20:49 on dist-test-slave-qsm9"
server_key: "97310bd2ab48c834b38e0869193134c3"
server_key_iv: "02a9e1a3e3657cdb6d1442d760b523e5"
server_key_version: "encryptionkey@0"
I20260926 20:20:49.971259 3097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:49.981315 3097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:49.984586 3097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:49.984784 3097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:49.985348 3097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:20:49.986148 3097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:49.986217 3097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:49.986290 3097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:49.986341 3097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:49.995141 3097 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:37099
I20260926 20:20:49.995182 3225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:37099 every 8 connection(s)
I20260926 20:20:49.996209 3097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:20:50.005622 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 3097
I20260926 20:20:50.005707 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
Sep 26 20:20:50 dist-test-slave-qsm9 krb5kdc[2856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:50.005985 467 external_mini_cluster.cc:1428] Setting key bd1b21f88162e21e99a42243331b1ee9
I20260926 20:20:50.010203 2940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:49.998165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:43529 (local address 127.0.116.254:38399)
0926 20:20:49.998326 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:49.998331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:49.999201 (+ 870us) server_negotiation.cc:407] Connection header received
0926 20:20:50.000643 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:50.000646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:50.000694 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:50.000769 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:50.001970 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:50.002512 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:50.003692 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:50.003907 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:50.003992 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:50.006317 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:50.006323 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:50.006330 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:50.006355 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:50.008146 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:50.008612 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:50.008619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:50.008623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:50.008716 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:50.009023 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:50.009029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:50.009034 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:50.009279 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:50.009369 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:50.009991 (+ 622us) server_negotiation.cc:299] Negotiation successful
0926 20:20:50.010089 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:20:50.011137 3228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:49.998599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:38399 (local address 127.0.116.194:43529)
0926 20:20:49.999045 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:49.999088 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:50.000447 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:50.000966 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:50.000972 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:50.001341 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:50.001812 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:50.001837 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:50.002645 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:50.002648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:50.003587 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:50.003596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:50.004040 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:50.004043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:50.004180 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:50.004697 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:50.004712 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:50.006160 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:50.008253 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:50.008257 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:50.008269 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:50.008516 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:50.008804 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:50.008806 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:50.008808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:50.008936 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:50.009396 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:50.009402 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:50.009716 (+ 314us) client_negotiation.cc:769] Sending connection context
0926 20:20:50.009963 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 20:20:50.010320 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
I20260926 20:20:50.012217 3226 heartbeater.cc:347] Connected to a master server at 127.0.116.254:38399
I20260926 20:20:50.012480 3226 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:50.012954 3226 heartbeater.cc:511] Master 127.0.116.254:38399 requested a full tablet report, sending...
I20260926 20:20:50.014008 2899 ts_manager.cc:194] Registered new tserver with Master: 5f80438615734ee2a5855ac807283a5b (127.0.116.194:37099)
I20260926 20:20:50.014581 2899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:43529
WARNING: no policy specified for kudu/127.0.116.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
I20260926 20:20:50.034528 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:38399
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:50.147184 3231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:50.147475 3231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:50.147542 3231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:50.151649 3231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:50.151710 3231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:50.151772 3231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:20:50.157150 3231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:38399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:50.158190 3231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:50.159163 3231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:50.169698 3239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:50.169684 3236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:50.169690 3237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:50.169870 3231 server_base.cc:1034] running on GCE node
I20260926 20:20:50.170463 3231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:50.171129 3231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:50.172303 3231 hybrid_clock.cc:648] HybridClock initialized: now 1790454050172285 us; error 29 us; skew 500 ppm
Sep 26 20:20:50 dist-test-slave-qsm9 krb5kdc[2856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:50.175137 3231 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:20:50.176122 3231 webserver.cc:460] Webserver started at http://127.0.116.195:40161/ using document root <none> and password file <none>
I20260926 20:20:50.176685 3231 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:50.176743 3231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:50.176954 3231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:50.178947 3231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "4d1b73f4419a45bdb1e2b8193344ea54"
format_stamp: "Formatted at 2026-09-26 20:20:50 on dist-test-slave-qsm9"
server_key: "a640f4c6524c639db8698daf11ce5926"
server_key_iv: "e22e6b6a8e3564f43883143efe73d73f"
server_key_version: "encryptionkey@0"
I20260926 20:20:50.179562 3231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "4d1b73f4419a45bdb1e2b8193344ea54"
format_stamp: "Formatted at 2026-09-26 20:20:50 on dist-test-slave-qsm9"
server_key: "a640f4c6524c639db8698daf11ce5926"
server_key_iv: "e22e6b6a8e3564f43883143efe73d73f"
server_key_version: "encryptionkey@0"
I20260926 20:20:50.183647 3231 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:20:50.185985 3246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:50.187274 3231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:20:50.187422 3231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "4d1b73f4419a45bdb1e2b8193344ea54"
format_stamp: "Formatted at 2026-09-26 20:20:50 on dist-test-slave-qsm9"
server_key: "a640f4c6524c639db8698daf11ce5926"
server_key_iv: "e22e6b6a8e3564f43883143efe73d73f"
server_key_version: "encryptionkey@0"
I20260926 20:20:50.187592 3231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:50.201849 3231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:50.204939 3231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:50.205152 3231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:50.205773 3231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:20:50.206632 3231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:50.206705 3231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:50.206779 3231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:50.206823 3231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:50.215934 3231 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:34983
I20260926 20:20:50.215916 3359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:34983 every 8 connection(s)
I20260926 20:20:50.216921 3231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:20:50.221220 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 3231
I20260926 20:20:50.221344 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:20:50.221691 467 external_mini_cluster.cc:1428] Setting key 8c6adeec786649b79243a7853be4730c
Sep 26 20:20:50 dist-test-slave-qsm9 krb5kdc[2856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:50.230923 2940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:50.218815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:55749 (local address 127.0.116.254:38399)
0926 20:20:50.218987 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:50.218992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:50.219912 (+ 920us) server_negotiation.cc:407] Connection header received
0926 20:20:50.221272 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:50.221279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:50.221343 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:50.221446 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:50.223023 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:50.223619 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:50.224807 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:50.225022 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:50.225100 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:50.227426 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:50.227429 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:50.227432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:50.227455 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:50.228990 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:50.229446 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:50.229448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:50.229450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:50.229522 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:50.229803 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:50.229805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:50.229806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:50.230002 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:50.230118 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:50.230636 (+ 518us) server_negotiation.cc:299] Negotiation successful
0926 20:20:50.230757 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:20:50.231856 3362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:50.219236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:38399 (local address 127.0.116.195:55749)
0926 20:20:50.219761 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:50.219805 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:50.221049 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:50.221678 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:50.221690 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:50.222303 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:50.222874 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:50.222887 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:50.223741 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:50.223745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:50.224698 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:50.224707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:50.225146 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:50.225149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:50.225293 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:50.225810 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:50.225826 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:50.227273 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:50.229088 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:50.229092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:50.229105 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:50.229353 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:50.229597 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:50.229599 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:50.229601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:50.229730 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:50.230117 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:50.230125 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:50.230393 (+ 268us) client_negotiation.cc:769] Sending connection context
0926 20:20:50.230597 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 20:20:50.230953 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":126,"threads_started":1}
I20260926 20:20:50.233016 3360 heartbeater.cc:347] Connected to a master server at 127.0.116.254:38399
I20260926 20:20:50.233271 3360 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:50.233784 3360 heartbeater.cc:511] Master 127.0.116.254:38399 requested a full tablet report, sending...
I20260926 20:20:50.234977 2899 ts_manager.cc:194] Registered new tserver with Master: 4d1b73f4419a45bdb1e2b8193344ea54 (127.0.116.195:34983)
I20260926 20:20:50.235790 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:20:50.235785 2899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:55749
W20260926 20:20:50.249218 2940 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:20:50.247778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32818 (local address 127.0.116.254:38399)
0926 20:20:50.247962 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:50.247965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:50.248099 (+ 134us) server_negotiation.cc:407] Connection header received
0926 20:20:50.248203 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:50.248205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:50.248250 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:50.248339 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:50.249047 (+ 708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32818: BlockingRecv error: recv got EOF from 127.0.0.1:32818 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 20:20:50.250368 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 2963
I20260926 20:20:50.256393 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 3097
I20260926 20:20:50.261811 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 3231
I20260926 20:20:50.266979 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 2869
2026-09-26T20:20:50Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2200 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:50 dist-test-slave-qsm9 krb5kdc[3372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:50 dist-test-slave-qsm9 krb5kdc[3372](info): set up 2 sockets
Sep 26 20:20:50 dist-test-slave-qsm9 krb5kdc[3372](info): commencing operation
krb5kdc: starting...
W20260926 20:20:51.311911 467 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:20:51 dist-test-slave-qsm9 krb5kdc[3372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:20:51.440855 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:45183
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:46267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:45183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:51.551987 3385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:51.552265 3385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:51.552305 3385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:51.556290 3385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:51.556353 3385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:51.556372 3385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:51.556389 3385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:51.556406 3385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:51.561609 3385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:46267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:45183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:45183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:51.562690 3385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:51.563696 3385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:51.572461 3391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:51.572453 3390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:51.572449 3393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:51.572525 3385 server_base.cc:1034] running on GCE node
I20260926 20:20:51.573360 3385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:51.574549 3385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:51.575709 3385 hybrid_clock.cc:648] HybridClock initialized: now 1790454051575698 us; error 28 us; skew 500 ppm
Sep 26 20:20:51 dist-test-slave-qsm9 krb5kdc[3372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:51.584327 3385 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:20:51.585346 3385 webserver.cc:460] Webserver started at http://127.0.116.254:43961/ using document root <none> and password file <none>
I20260926 20:20:51.585959 3385 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:51.586027 3385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:51.586269 3385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:51.588619 3385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "2468d132d8804b1f8a1b8c4f993b8764"
format_stamp: "Formatted at 2026-09-26 20:20:51 on dist-test-slave-qsm9"
server_key: "6ca24d2243c53b5a75f3980e9d33158f"
server_key_iv: "d198473783f1237fd1324125128238ab"
server_key_version: "encryptionkey@0"
I20260926 20:20:51.589207 3385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "2468d132d8804b1f8a1b8c4f993b8764"
format_stamp: "Formatted at 2026-09-26 20:20:51 on dist-test-slave-qsm9"
server_key: "6ca24d2243c53b5a75f3980e9d33158f"
server_key_iv: "d198473783f1237fd1324125128238ab"
server_key_version: "encryptionkey@0"
I20260926 20:20:51.593098 3385 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:20:51.595459 3400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:51.596853 3385 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:20:51.596969 3385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "2468d132d8804b1f8a1b8c4f993b8764"
format_stamp: "Formatted at 2026-09-26 20:20:51 on dist-test-slave-qsm9"
server_key: "6ca24d2243c53b5a75f3980e9d33158f"
server_key_iv: "d198473783f1237fd1324125128238ab"
server_key_version: "encryptionkey@0"
I20260926 20:20:51.597069 3385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:51.613559 3385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:51.616698 3385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:51.616909 3385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:51.624137 3385 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:45183
I20260926 20:20:51.624468 3452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:45183 every 8 connection(s)
I20260926 20:20:51.625138 3385 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:51.627317 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 3385
I20260926 20:20:51.627424 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:20:51.627739 467 external_mini_cluster.cc:1428] Setting key 4688670869ef11705fd9b224b7193fa5
I20260926 20:20:51.628095 3453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:51.635285 3453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764: Bootstrap starting.
Sep 26 20:20:51 dist-test-slave-qsm9 krb5kdc[3372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:51.637818 3453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:51.638751 3453 log.cc:824] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:51.640800 3453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764: No bootstrap required, opened a new log
I20260926 20:20:51.642725 3456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:51.629154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60932 (local address 127.0.116.254:45183)
0926 20:20:51.629605 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:51.629625 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:51.629666 (+ 41us) server_negotiation.cc:407] Connection header received
0926 20:20:51.630448 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:51.630472 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:51.630914 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:51.631309 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:51.632045 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:51.633517 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:51.634360 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:51.635023 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:51.635181 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:51.637856 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:51.637865 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:51.637884 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:51.637917 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:51.639802 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:51.640282 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:51.640287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:51.640292 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:51.640412 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:51.640713 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:51.640717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:51.640718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:51.641186 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:51.641400 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:51.641684 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 20:20:51.641991 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":109,"threads_started":1}
I20260926 20:20:51.644863 3453 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2468d132d8804b1f8a1b8c4f993b8764" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45183 } }
I20260926 20:20:51.645112 3453 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:51.645191 3453 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2468d132d8804b1f8a1b8c4f993b8764, State: Initialized, Role: FOLLOWER
I20260926 20:20:51.645673 3453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2468d132d8804b1f8a1b8c4f993b8764" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45183 } }
I20260926 20:20:51.645833 3453 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:51.645957 3453 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:51.645968 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:20:51.646085 3453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:51.647161 3453 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2468d132d8804b1f8a1b8c4f993b8764" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45183 } }
I20260926 20:20:51.647648 3453 leader_election.cc:302] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2468d132d8804b1f8a1b8c4f993b8764; no voters:
I20260926 20:20:51.647979 3453 leader_election.cc:288] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:51.648141 3458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:51.648486 3458 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764 [term 1 LEADER]: Becoming Leader. State: Replica: 2468d132d8804b1f8a1b8c4f993b8764, State: Running, Role: LEADER
I20260926 20:20:51.648870 3458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2468d132d8804b1f8a1b8c4f993b8764" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45183 } }
I20260926 20:20:51.649499 3453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:51.650151 3458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2468d132d8804b1f8a1b8c4f993b8764. Latest consensus state: current_term: 1 leader_uuid: "2468d132d8804b1f8a1b8c4f993b8764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2468d132d8804b1f8a1b8c4f993b8764" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45183 } } }
I20260926 20:20:51.650388 3458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:51.650586 3460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2468d132d8804b1f8a1b8c4f993b8764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2468d132d8804b1f8a1b8c4f993b8764" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45183 } } }
I20260926 20:20:51.650754 3460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:51.650852 3466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:51.653859 3466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:51.659773 3466 catalog_manager.cc:1347] Generated new cluster ID: 0e45df57cc824a8f87a25da2197e8dc0
I20260926 20:20:51.659909 3466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:51.672796 3456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:51.667795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60938 (local address 127.0.116.254:45183)
0926 20:20:51.667935 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:51.667939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:51.668197 (+ 258us) server_negotiation.cc:407] Connection header received
0926 20:20:51.668367 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:51.668371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:51.668425 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:51.668517 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:51.669057 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:51.669623 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:51.670483 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:51.670727 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:51.670839 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:51.671803 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:51.671806 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:51.671820 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:51.672338 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:51.672368 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:51.672384 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:51.672488 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:51.672569 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 20:20:51.672629 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:20:51.676112 3466 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:51.677309 3466 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:51.685735 3466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2468d132d8804b1f8a1b8c4f993b8764: Generated new TSK 0
I20260926 20:20:51.686679 3466 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:20:51.699936 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 3385
2026-09-26T20:20:51Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1430 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:51 dist-test-slave-qsm9 krb5kdc[3486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:51 dist-test-slave-qsm9 krb5kdc[3486](info): set up 2 sockets
Sep 26 20:20:51 dist-test-slave-qsm9 krb5kdc[3486](info): commencing operation
krb5kdc: starting...
W20260926 20:20:52.735831 467 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:20:52 dist-test-slave-qsm9 krb5kdc[3486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:20:52.862924 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:46395
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:46171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:46395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:52.979106 3499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:52.979436 3499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:52.979512 3499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:52.983538 3499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:52.983636 3499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:52.983676 3499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:52.983736 3499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:52.983774 3499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:52.989257 3499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:46171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:46395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:46395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:52.990399 3499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:52.991475 3499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:53.000758 3507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:53.000758 3504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:53.000933 3499 server_base.cc:1034] running on GCE node
W20260926 20:20:53.000808 3505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:53.001622 3499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:53.002882 3499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:53.004094 3499 hybrid_clock.cc:648] HybridClock initialized: now 1790454053004078 us; error 27 us; skew 500 ppm
Sep 26 20:20:53 dist-test-slave-qsm9 krb5kdc[3486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:53.007074 3499 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:20:53.008116 3499 webserver.cc:460] Webserver started at http://127.0.116.254:45575/ using document root <none> and password file <none>
I20260926 20:20:53.008785 3499 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:53.008879 3499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:53.009121 3499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:53.011228 3499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "85df37daff744231af0869cfb6ccdc1b"
format_stamp: "Formatted at 2026-09-26 20:20:53 on dist-test-slave-qsm9"
server_key: "52f71ac39c31c2779a3b5f32e3e5046b"
server_key_iv: "982ce87626cb5220eba7b057bf7c3724"
server_key_version: "encryptionkey@0"
I20260926 20:20:53.011823 3499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "85df37daff744231af0869cfb6ccdc1b"
format_stamp: "Formatted at 2026-09-26 20:20:53 on dist-test-slave-qsm9"
server_key: "52f71ac39c31c2779a3b5f32e3e5046b"
server_key_iv: "982ce87626cb5220eba7b057bf7c3724"
server_key_version: "encryptionkey@0"
I20260926 20:20:53.015604 3499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:20:53.018095 3514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:53.019387 3499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:20:53.019547 3499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "85df37daff744231af0869cfb6ccdc1b"
format_stamp: "Formatted at 2026-09-26 20:20:53 on dist-test-slave-qsm9"
server_key: "52f71ac39c31c2779a3b5f32e3e5046b"
server_key_iv: "982ce87626cb5220eba7b057bf7c3724"
server_key_version: "encryptionkey@0"
I20260926 20:20:53.019665 3499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:53.038229 3499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:53.041617 3499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:53.041836 3499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:53.049365 3499 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:46395
I20260926 20:20:53.049714 3566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:46395 every 8 connection(s)
I20260926 20:20:53.050467 3499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:53.053543 3567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:53.059322 3567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b: Bootstrap starting.
I20260926 20:20:53.059669 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 3499
I20260926 20:20:53.059798 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:20:53.060084 467 external_mini_cluster.cc:1428] Setting key 78dd30e9b61be85db0117518c9cf2e41
I20260926 20:20:53.062352 3567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:53.063155 3567 log.cc:824] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:53.065258 3567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b: No bootstrap required, opened a new log
Sep 26 20:20:53 dist-test-slave-qsm9 krb5kdc[3486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:53.069424 3567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85df37daff744231af0869cfb6ccdc1b" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46395 } }
I20260926 20:20:53.069727 3567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:53.069819 3567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85df37daff744231af0869cfb6ccdc1b, State: Initialized, Role: FOLLOWER
I20260926 20:20:53.070293 3567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85df37daff744231af0869cfb6ccdc1b" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46395 } }
I20260926 20:20:53.070462 3567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:53.070562 3567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:53.070675 3567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:53.071952 3567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85df37daff744231af0869cfb6ccdc1b" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46395 } }
I20260926 20:20:53.072348 3567 leader_election.cc:302] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85df37daff744231af0869cfb6ccdc1b; no voters:
I20260926 20:20:53.072710 3567 leader_election.cc:288] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:53.072789 3572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:53.073037 3572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b [term 1 LEADER]: Becoming Leader. State: Replica: 85df37daff744231af0869cfb6ccdc1b, State: Running, Role: LEADER
I20260926 20:20:53.073455 3572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85df37daff744231af0869cfb6ccdc1b" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46395 } }
I20260926 20:20:53.074187 3567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:53.075649 3574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85df37daff744231af0869cfb6ccdc1b. Latest consensus state: current_term: 1 leader_uuid: "85df37daff744231af0869cfb6ccdc1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85df37daff744231af0869cfb6ccdc1b" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46395 } } }
I20260926 20:20:53.075783 3574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:53.076040 3573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85df37daff744231af0869cfb6ccdc1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85df37daff744231af0869cfb6ccdc1b" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46395 } } }
I20260926 20:20:53.076174 3573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:53.076202 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:53.061515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35910 (local address 127.0.116.254:46395)
0926 20:20:53.062020 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:53.062036 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:53.062074 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:20:53.062807 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:53.062827 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:53.063173 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:53.063599 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:53.064458 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:53.065539 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:53.066412 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:53.066918 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:53.067096 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:53.070387 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:53.070393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:53.070406 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:53.070437 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:53.072529 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:53.073140 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:53.073162 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:53.073168 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:53.073316 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:53.073669 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:53.073673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:53.073675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:53.074115 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:53.074344 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:53.074655 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 20:20:53.075030 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":152,"threads_started":1}
I20260926 20:20:53.076310 3581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:53.080080 3581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:53.086375 3581 catalog_manager.cc:1347] Generated new cluster ID: c662e15afd5a42ffa5e5b5b3be3e39c3
I20260926 20:20:53.086566 3581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:53.097931 3581 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:53.099270 3581 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:53.109799 3581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85df37daff744231af0869cfb6ccdc1b: Generated new TSK 0
I20260926 20:20:53.110692 3581 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
I20260926 20:20:53.147817 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46395
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:46171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:53.279255 3593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:53.279578 3593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:53.279624 3593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:53.283715 3593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:53.283784 3593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:53.283850 3593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:20:53.289266 3593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:46171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:53.290427 3593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:53.291536 3593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:53.301888 3599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:53.302050 3593 server_base.cc:1034] running on GCE node
W20260926 20:20:53.301906 3598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:53.301910 3601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:53.302606 3593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:53.303300 3593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:53.304490 3593 hybrid_clock.cc:648] HybridClock initialized: now 1790454053304480 us; error 62 us; skew 500 ppm
Sep 26 20:20:53 dist-test-slave-qsm9 krb5kdc[3486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:53.307153 3593 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:20:53.308071 3593 webserver.cc:460] Webserver started at http://127.0.116.193:35043/ using document root <none> and password file <none>
I20260926 20:20:53.308634 3593 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:53.308722 3593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:53.308954 3593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:53.310930 3593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "0abe118d20b346d0968699a9ff0869be"
format_stamp: "Formatted at 2026-09-26 20:20:53 on dist-test-slave-qsm9"
server_key: "4a5faf4ea925d0fb0ced99187241f3b2"
server_key_iv: "396e5e3b2d6306c61f6a03d83346e1d9"
server_key_version: "encryptionkey@0"
I20260926 20:20:53.311486 3593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "0abe118d20b346d0968699a9ff0869be"
format_stamp: "Formatted at 2026-09-26 20:20:53 on dist-test-slave-qsm9"
server_key: "4a5faf4ea925d0fb0ced99187241f3b2"
server_key_iv: "396e5e3b2d6306c61f6a03d83346e1d9"
server_key_version: "encryptionkey@0"
I20260926 20:20:53.315074 3593 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:20:53.317467 3608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:53.318672 3593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:20:53.318804 3593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "0abe118d20b346d0968699a9ff0869be"
format_stamp: "Formatted at 2026-09-26 20:20:53 on dist-test-slave-qsm9"
server_key: "4a5faf4ea925d0fb0ced99187241f3b2"
server_key_iv: "396e5e3b2d6306c61f6a03d83346e1d9"
server_key_version: "encryptionkey@0"
I20260926 20:20:53.318912 3593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:53.339476 3593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:53.342931 3593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:53.343155 3593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:53.343859 3593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:20:53.344707 3593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:53.344753 3593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:53.344825 3593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:53.344854 3593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:53.354056 3593 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:44147
I20260926 20:20:53.354074 3721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:44147 every 8 connection(s)
I20260926 20:20:53.355058 3593 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:20:53.355646 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 3593
I20260926 20:20:53.355760 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:20:53.356065 467 external_mini_cluster.cc:1428] Setting key 60758564830ffad126c7b332586bd998
WARNING: no policy specified for kudu/127.0.116.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.194@KRBTEST.COM" created.
Sep 26 20:20:53 dist-test-slave-qsm9 krb5kdc[3486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:53.375361 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:53.357110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:44031 (local address 127.0.116.254:46395)
0926 20:20:53.357261 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:53.357266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:53.358280 (+ 1014us) server_negotiation.cc:407] Connection header received
0926 20:20:53.360134 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:53.360137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:53.360186 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:53.360264 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:53.362005 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:53.362604 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:53.364583 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:53.364821 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:53.364927 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:53.370985 (+ 6058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:53.370988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:53.370990 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:53.371021 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:53.372803 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:53.373411 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:53.373421 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:53.373423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:53.373509 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:53.374020 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:53.374023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:53.374027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:53.374236 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:53.374369 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:53.375030 (+ 661us) server_negotiation.cc:299] Negotiation successful
0926 20:20:53.375170 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 20:20:53.376274 3724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:53.357610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46395 (local address 127.0.116.193:44031)
0926 20:20:53.358112 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:53.358165 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:53.359857 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:53.360468 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:53.360481 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:53.361037 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:53.361771 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:53.361796 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:53.362758 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:53.362764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:53.364427 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:53.364439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:53.365009 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:53.365015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:53.365254 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:53.365926 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:53.365957 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:53.370729 (+ 4772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:53.372949 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:53.372955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:53.372970 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:53.373268 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:53.373619 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:53.373626 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:53.373628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:53.373863 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:53.374379 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:53.374389 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:53.374667 (+ 278us) client_negotiation.cc:769] Sending connection context
0926 20:20:53.374915 (+ 248us) client_negotiation.cc:240] Negotiation successful
0926 20:20:53.375305 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":139,"threads_started":1}
I20260926 20:20:53.377775 3722 heartbeater.cc:347] Connected to a master server at 127.0.116.254:46395
I20260926 20:20:53.378082 3722 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:53.378717 3722 heartbeater.cc:511] Master 127.0.116.254:46395 requested a full tablet report, sending...
I20260926 20:20:53.380488 3531 ts_manager.cc:194] Registered new tserver with Master: 0abe118d20b346d0968699a9ff0869be (127.0.116.193:44147)
I20260926 20:20:53.381846 3531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:44031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
I20260926 20:20:53.386933 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46395
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:46171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:53.505780 3727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:53.506065 3727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:53.506108 3727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:53.510340 3727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:53.510417 3727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:53.510479 3727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:20:53.515947 3727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:46171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:53.517009 3727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:53.517982 3727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:53.528208 3732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:53.528198 3733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:53.528198 3735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:53.528293 3727 server_base.cc:1034] running on GCE node
I20260926 20:20:53.528971 3727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:53.529614 3727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:53.530802 3727 hybrid_clock.cc:648] HybridClock initialized: now 1790454053530783 us; error 30 us; skew 500 ppm
Sep 26 20:20:53 dist-test-slave-qsm9 krb5kdc[3486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:53.533605 3727 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:20:53.534482 3727 webserver.cc:460] Webserver started at http://127.0.116.194:41269/ using document root <none> and password file <none>
I20260926 20:20:53.535043 3727 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:53.535133 3727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:53.535363 3727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:53.537468 3727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "93c1830a77f54804aea3aac77791187c"
format_stamp: "Formatted at 2026-09-26 20:20:53 on dist-test-slave-qsm9"
server_key: "ab17df0dfe079f2fc8ebea1db77b2d50"
server_key_iv: "d0cf9d1f5ffbc7570447f61e26783943"
server_key_version: "encryptionkey@0"
I20260926 20:20:53.538044 3727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "93c1830a77f54804aea3aac77791187c"
format_stamp: "Formatted at 2026-09-26 20:20:53 on dist-test-slave-qsm9"
server_key: "ab17df0dfe079f2fc8ebea1db77b2d50"
server_key_iv: "d0cf9d1f5ffbc7570447f61e26783943"
server_key_version: "encryptionkey@0"
I20260926 20:20:53.541781 3727 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 20:20:53.544333 3742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:53.545663 3727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:20:53.545809 3727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "93c1830a77f54804aea3aac77791187c"
format_stamp: "Formatted at 2026-09-26 20:20:53 on dist-test-slave-qsm9"
server_key: "ab17df0dfe079f2fc8ebea1db77b2d50"
server_key_iv: "d0cf9d1f5ffbc7570447f61e26783943"
server_key_version: "encryptionkey@0"
I20260926 20:20:53.545948 3727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:53.570405 3727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:53.573781 3727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:53.574011 3727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:53.574728 3727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:20:53.575727 3727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:53.575805 3727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:53.575902 3727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:53.575955 3727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:53.584748 3727 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:35757
I20260926 20:20:53.584779 3855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:35757 every 8 connection(s)
I20260926 20:20:53.585632 3727 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:20:53.593532 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 3727
I20260926 20:20:53.593668 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:20:53.594058 467 external_mini_cluster.cc:1428] Setting key 813df527d42db505e2c1c0379d51077a
Sep 26 20:20:53 dist-test-slave-qsm9 krb5kdc[3486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:53.599735 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:53.587445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:60603 (local address 127.0.116.254:46395)
0926 20:20:53.587623 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:53.587627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:53.588469 (+ 842us) server_negotiation.cc:407] Connection header received
0926 20:20:53.589818 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:53.589821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:53.589873 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:53.589950 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:53.591099 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:53.591710 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:53.592811 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:53.593036 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:53.593114 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:53.595898 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:53.595901 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:53.595904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:53.595930 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:53.597683 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:53.598191 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:53.598195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:53.598197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:53.598285 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:53.598624 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:53.598629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:53.598631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:53.598820 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:53.598937 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:53.599441 (+ 504us) server_negotiation.cc:299] Negotiation successful
0926 20:20:53.599578 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 20:20:53.600672 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:53.587875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46395 (local address 127.0.116.194:60603)
0926 20:20:53.588305 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:53.588357 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:53.589612 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:53.590112 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:53.590119 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:53.590468 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:53.590968 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:53.590981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:53.591813 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:53.591816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:53.592708 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:53.592716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:53.593163 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:53.593165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:53.593314 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:53.593963 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:53.593986 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:53.595739 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:53.597802 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:53.597809 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:53.597824 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:53.598081 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:53.598381 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:53.598385 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:53.598387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:53.598527 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:53.598933 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:53.598940 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:53.599194 (+ 254us) client_negotiation.cc:769] Sending connection context
0926 20:20:53.599399 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 20:20:53.599749 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":109,"threads_started":1}
I20260926 20:20:53.602056 3856 heartbeater.cc:347] Connected to a master server at 127.0.116.254:46395
I20260926 20:20:53.602355 3856 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:53.603009 3856 heartbeater.cc:511] Master 127.0.116.254:46395 requested a full tablet report, sending...
I20260926 20:20:53.604367 3531 ts_manager.cc:194] Registered new tserver with Master: 93c1830a77f54804aea3aac77791187c (127.0.116.194:35757)
WARNING: no policy specified for kudu/127.0.116.195@KRBTEST.COM; defaulting to no policy
I20260926 20:20:53.605257 3531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:60603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
I20260926 20:20:53.622895 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46395
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:46171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:53.733306 3861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:53.733600 3861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:53.733646 3861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:53.737810 3861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:53.737886 3861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:53.737948 3861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:20:53.743335 3861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:46171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:53.744545 3861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:53.745573 3861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:53.755872 3866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:53.755874 3867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:53.755970 3869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:53.755957 3861 server_base.cc:1034] running on GCE node
I20260926 20:20:53.756577 3861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:53.757295 3861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:53.758467 3861 hybrid_clock.cc:648] HybridClock initialized: now 1790454053758448 us; error 32 us; skew 500 ppm
Sep 26 20:20:53 dist-test-slave-qsm9 krb5kdc[3486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:53.761410 3861 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:20:53.762326 3861 webserver.cc:460] Webserver started at http://127.0.116.195:43873/ using document root <none> and password file <none>
I20260926 20:20:53.762872 3861 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:53.762943 3861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:53.763120 3861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:53.765296 3861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "f96ab4152af44d20bb486acf7575d51d"
format_stamp: "Formatted at 2026-09-26 20:20:53 on dist-test-slave-qsm9"
server_key: "335b7c2aeafe6142ff24b1ad13729344"
server_key_iv: "5a54ddc48baea4f7b7f4de444535fa78"
server_key_version: "encryptionkey@0"
I20260926 20:20:53.765807 3861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "f96ab4152af44d20bb486acf7575d51d"
format_stamp: "Formatted at 2026-09-26 20:20:53 on dist-test-slave-qsm9"
server_key: "335b7c2aeafe6142ff24b1ad13729344"
server_key_iv: "5a54ddc48baea4f7b7f4de444535fa78"
server_key_version: "encryptionkey@0"
I20260926 20:20:53.769690 3861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 20:20:53.772326 3876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:53.774173 3861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:20:53.774279 3861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "f96ab4152af44d20bb486acf7575d51d"
format_stamp: "Formatted at 2026-09-26 20:20:53 on dist-test-slave-qsm9"
server_key: "335b7c2aeafe6142ff24b1ad13729344"
server_key_iv: "5a54ddc48baea4f7b7f4de444535fa78"
server_key_version: "encryptionkey@0"
I20260926 20:20:53.774374 3861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:53.787361 3861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:53.790970 3861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:53.791144 3861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:53.791759 3861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:20:53.792549 3861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:53.792595 3861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:53.792634 3861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:53.792649 3861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:53.801928 3861 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:37631
I20260926 20:20:53.801968 3989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:37631 every 8 connection(s)
I20260926 20:20:53.802834 3861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:20:53.809605 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 3861
I20260926 20:20:53.809772 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:20:53.810153 467 external_mini_cluster.cc:1428] Setting key 19715600c0d44b68d50e9b873958b96e
Sep 26 20:20:53 dist-test-slave-qsm9 krb5kdc[3486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:53.818192 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:53.804778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:51641 (local address 127.0.116.254:46395)
0926 20:20:53.804942 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:53.804947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:53.805821 (+ 874us) server_negotiation.cc:407] Connection header received
0926 20:20:53.807225 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:53.807228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:53.807279 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:53.807374 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:53.808735 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:53.809439 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:53.811207 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:53.811486 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:53.811611 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:53.814337 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:53.814340 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:53.814341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:53.814371 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:53.815966 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:53.816529 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:53.816532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:53.816533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:53.816612 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:53.817086 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:53.817090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:53.817091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:53.817332 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:53.817428 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:53.817956 (+ 528us) server_negotiation.cc:299] Negotiation successful
0926 20:20:53.818068 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 20:20:53.819103 3992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:53.805191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46395 (local address 127.0.116.195:51641)
0926 20:20:53.805667 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:53.805711 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:53.807001 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:53.807620 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:53.807628 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:53.808058 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:53.808579 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:53.808592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:53.809598 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:53.809602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:53.811078 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:53.811093 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:53.811671 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:53.811678 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:53.811886 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:53.812605 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:53.812625 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:53.814161 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:53.816097 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:53.816103 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:53.816116 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:53.816413 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:53.816697 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:53.816699 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:53.816701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:53.816865 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:53.817436 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:53.817442 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:53.817709 (+ 267us) client_negotiation.cc:769] Sending connection context
0926 20:20:53.817896 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 20:20:53.818185 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":111,"threads_started":1}
I20260926 20:20:53.820524 3990 heartbeater.cc:347] Connected to a master server at 127.0.116.254:46395
I20260926 20:20:53.820820 3990 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:53.821412 3990 heartbeater.cc:511] Master 127.0.116.254:46395 requested a full tablet report, sending...
I20260926 20:20:53.822559 3531 ts_manager.cc:194] Registered new tserver with Master: f96ab4152af44d20bb486acf7575d51d (127.0.116.195:37631)
I20260926 20:20:53.823473 3531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:51641
I20260926 20:20:53.824776 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:20:53 dist-test-slave-qsm9 krb5kdc[3486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454053, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 20:20:53 dist-test-slave-qsm9 krb5kdc[3486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454053, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:53.851080 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:53.842299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.0.116.254:46395)
0926 20:20:53.842457 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:53.842463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:53.842615 (+ 152us) server_negotiation.cc:407] Connection header received
0926 20:20:53.842728 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:53.842731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:53.842775 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:53.842844 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:53.843614 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:53.844135 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:53.844978 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:53.845193 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:53.845287 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:53.847915 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:53.847918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:53.847920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:53.847946 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:53.849411 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:53.849808 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:53.849811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:53.849812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:53.849886 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:53.850192 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:53.850194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:53.850195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:53.850428 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:53.850528 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:53.850787 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 20:20:53.850886 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 20:20:53.851589 3531 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35912
I20260926 20:20:53.852674 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 3593
I20260926 20:20:53.858610 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 3727
I20260926 20:20:53.864351 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 3861
I20260926 20:20:53.869305 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 3499
2026-09-26T20:20:53Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2171 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:53 dist-test-slave-qsm9 krb5kdc[4002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:53 dist-test-slave-qsm9 krb5kdc[4002](info): set up 2 sockets
Sep 26 20:20:53 dist-test-slave-qsm9 krb5kdc[4002](info): commencing operation
krb5kdc: starting...
W20260926 20:20:54.908154 467 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:20:54 dist-test-slave-qsm9 krb5kdc[4002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:20:55.035933 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:46015
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:38155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:46015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:55.148754 4015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:55.149096 4015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:55.149180 4015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:55.153264 4015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:55.153357 4015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:55.153395 4015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:55.153455 4015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:55.153493 4015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:55.159085 4015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:38155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:46015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:46015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:55.160285 4015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:55.161412 4015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:20:55.170349 4015 server_base.cc:1034] running on GCE node
W20260926 20:20:55.170207 4021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:55.170207 4023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:55.170320 4020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:55.171162 4015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:55.172415 4015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:55.173631 4015 hybrid_clock.cc:648] HybridClock initialized: now 1790454055173575 us; error 69 us; skew 500 ppm
Sep 26 20:20:55 dist-test-slave-qsm9 krb5kdc[4002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:55.176661 4015 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:20:55.177680 4015 webserver.cc:460] Webserver started at http://127.0.116.254:45253/ using document root <none> and password file <none>
I20260926 20:20:55.178267 4015 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:55.178363 4015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:55.178611 4015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:55.180806 4015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "589320438d9840aa91f5f440b38dc9aa"
format_stamp: "Formatted at 2026-09-26 20:20:55 on dist-test-slave-qsm9"
server_key: "52cf48ea0c6ee4821f84ce0f4e8c2790"
server_key_iv: "370a9e35fc1cdf68ea17865b13d231ee"
server_key_version: "encryptionkey@0"
I20260926 20:20:55.181427 4015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "589320438d9840aa91f5f440b38dc9aa"
format_stamp: "Formatted at 2026-09-26 20:20:55 on dist-test-slave-qsm9"
server_key: "52cf48ea0c6ee4821f84ce0f4e8c2790"
server_key_iv: "370a9e35fc1cdf68ea17865b13d231ee"
server_key_version: "encryptionkey@0"
I20260926 20:20:55.185448 4015 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:20:55.187994 4030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:55.189250 4015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:20:55.189395 4015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "589320438d9840aa91f5f440b38dc9aa"
format_stamp: "Formatted at 2026-09-26 20:20:55 on dist-test-slave-qsm9"
server_key: "52cf48ea0c6ee4821f84ce0f4e8c2790"
server_key_iv: "370a9e35fc1cdf68ea17865b13d231ee"
server_key_version: "encryptionkey@0"
I20260926 20:20:55.189509 4015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:55.212082 4015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:55.215165 4015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:55.215389 4015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:55.222781 4015 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:46015
I20260926 20:20:55.222811 4082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:46015 every 8 connection(s)
I20260926 20:20:55.223795 4015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:55.226694 4083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:55.232357 4083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa: Bootstrap starting.
I20260926 20:20:55.232949 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 4015
I20260926 20:20:55.233182 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:20:55.233461 467 external_mini_cluster.cc:1428] Setting key 78e562c02644cea835aee42564a60dba
I20260926 20:20:55.235235 4083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:55.236060 4083 log.cc:824] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:55.238255 4083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa: No bootstrap required, opened a new log
I20260926 20:20:55.242281 4083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589320438d9840aa91f5f440b38dc9aa" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46015 } }
I20260926 20:20:55.242534 4083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:55.242617 4083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589320438d9840aa91f5f440b38dc9aa, State: Initialized, Role: FOLLOWER
Sep 26 20:20:55 dist-test-slave-qsm9 krb5kdc[4002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:55.243180 4083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589320438d9840aa91f5f440b38dc9aa" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46015 } }
I20260926 20:20:55.243357 4083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:55.243438 4083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:55.243559 4083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:55.244509 4083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589320438d9840aa91f5f440b38dc9aa" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46015 } }
I20260926 20:20:55.244944 4083 leader_election.cc:302] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 589320438d9840aa91f5f440b38dc9aa; no voters:
I20260926 20:20:55.245325 4083 leader_election.cc:288] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:55.245431 4088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:55.245654 4088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa [term 1 LEADER]: Becoming Leader. State: Replica: 589320438d9840aa91f5f440b38dc9aa, State: Running, Role: LEADER
I20260926 20:20:55.246060 4088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589320438d9840aa91f5f440b38dc9aa" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46015 } }
I20260926 20:20:55.246759 4083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:55.247839 4089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "589320438d9840aa91f5f440b38dc9aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589320438d9840aa91f5f440b38dc9aa" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46015 } } }
I20260926 20:20:55.247982 4089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:55.248397 4090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 589320438d9840aa91f5f440b38dc9aa. Latest consensus state: current_term: 1 leader_uuid: "589320438d9840aa91f5f440b38dc9aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589320438d9840aa91f5f440b38dc9aa" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46015 } } }
I20260926 20:20:55.248562 4090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:55.249997 4086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:55.234862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52430 (local address 127.0.116.254:46015)
0926 20:20:55.235293 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:55.235308 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:55.235374 (+ 66us) server_negotiation.cc:407] Connection header received
0926 20:20:55.236103 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:55.236121 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:55.236517 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:55.236846 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:55.237524 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:55.238958 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:55.239861 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:55.240577 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:55.240751 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:55.243526 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:55.243537 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:55.243558 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:55.243597 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:55.245912 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:55.246416 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:55.246422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:55.246428 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:55.246586 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:55.246939 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:55.246943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:55.246944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:55.247527 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:55.248521 (+ 994us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:55.248687 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 20:20:55.249202 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":146,"threads_started":1}
W20260926 20:20:55.252887 4103 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:20:55.252950 4103 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:20:55.253028 4097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:55.253746 4097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:55.258929 4097 catalog_manager.cc:1347] Generated new cluster ID: a71aef66568f4609837aaf5a8eb3430c
I20260926 20:20:55.259001 4097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:55.270753 4097 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:55.272100 4097 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:55.279548 4097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 589320438d9840aa91f5f440b38dc9aa: Generated new TSK 0
I20260926 20:20:55.280346 4097 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
I20260926 20:20:55.311076 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46015
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:38155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:55.428102 4109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:55.428370 4109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:55.428440 4109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:55.432526 4109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:55.432587 4109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:55.432677 4109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:20:55.438087 4109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:38155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:55.439184 4109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:55.440303 4109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:55.450589 4115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:55.450577 4114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:55.450583 4117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:55.450924 4109 server_base.cc:1034] running on GCE node
I20260926 20:20:55.451323 4109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:55.452049 4109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:55.453248 4109 hybrid_clock.cc:648] HybridClock initialized: now 1790454055453218 us; error 48 us; skew 500 ppm
Sep 26 20:20:55 dist-test-slave-qsm9 krb5kdc[4002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:55.456324 4109 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:20:55.457221 4109 webserver.cc:460] Webserver started at http://127.0.116.193:35355/ using document root <none> and password file <none>
I20260926 20:20:55.457764 4109 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:55.457821 4109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:55.458014 4109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:55.460202 4109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "e3a169a4829a4b769fb2936a778d88bd"
format_stamp: "Formatted at 2026-09-26 20:20:55 on dist-test-slave-qsm9"
server_key: "07eae4966181f80596b70c1fdfd71829"
server_key_iv: "e9d904d69415320a7cd800da6be05e1b"
server_key_version: "encryptionkey@0"
I20260926 20:20:55.460783 4109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "e3a169a4829a4b769fb2936a778d88bd"
format_stamp: "Formatted at 2026-09-26 20:20:55 on dist-test-slave-qsm9"
server_key: "07eae4966181f80596b70c1fdfd71829"
server_key_iv: "e9d904d69415320a7cd800da6be05e1b"
server_key_version: "encryptionkey@0"
I20260926 20:20:55.464830 4109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:20:55.467592 4124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:55.468924 4109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:20:55.469030 4109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "e3a169a4829a4b769fb2936a778d88bd"
format_stamp: "Formatted at 2026-09-26 20:20:55 on dist-test-slave-qsm9"
server_key: "07eae4966181f80596b70c1fdfd71829"
server_key_iv: "e9d904d69415320a7cd800da6be05e1b"
server_key_version: "encryptionkey@0"
I20260926 20:20:55.469177 4109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:55.496002 4109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:55.499543 4109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:55.499763 4109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:55.500388 4109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:20:55.501220 4109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:55.501266 4109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:55.501333 4109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:55.501374 4109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:55.510111 4109 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:38439
I20260926 20:20:55.510133 4237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:38439 every 8 connection(s)
I20260926 20:20:55.511031 4109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:20:55.518075 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 4109
I20260926 20:20:55.518188 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:20:55.518486 467 external_mini_cluster.cc:1428] Setting key 2dc0cebc4babd22fbc9d2635f5fd3203
Sep 26 20:20:55 dist-test-slave-qsm9 krb5kdc[4002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:55.525301 4086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:55.513047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:59029 (local address 127.0.116.254:46015)
0926 20:20:55.513205 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:55.513209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:55.514146 (+ 937us) server_negotiation.cc:407] Connection header received
0926 20:20:55.515634 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:55.515637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:55.515698 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:55.515807 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:55.517032 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:55.517572 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:55.518896 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:55.519141 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:55.519253 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:55.521702 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:55.521710 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:55.521712 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:55.521736 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:55.523142 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:55.523736 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:55.523739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:55.523741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:55.523825 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:55.524175 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:55.524178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:55.524180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:55.524382 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:55.524490 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:55.525003 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 20:20:55.525126 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 20:20:55.526098 4240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:55.513456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46015 (local address 127.0.116.193:59029)
0926 20:20:55.513979 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:55.514023 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:55.515372 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:55.515995 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:55.516003 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:55.516364 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:55.516848 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:55.516861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:55.517701 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:55.517704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:55.518765 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:55.518774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:55.519293 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:55.519297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:55.519479 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:55.520107 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:55.520122 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:55.521535 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:55.523263 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:55.523269 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:55.523284 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:55.523624 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:55.523922 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:55.523927 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:55.523929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:55.524074 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:55.524489 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:55.524497 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:55.524751 (+ 254us) client_negotiation.cc:769] Sending connection context
0926 20:20:55.524947 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 20:20:55.525269 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":103,"threads_started":1}
I20260926 20:20:55.527419 4238 heartbeater.cc:347] Connected to a master server at 127.0.116.254:46015
I20260926 20:20:55.527719 4238 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.116.194@KRBTEST.COM; defaulting to no policy
I20260926 20:20:55.528286 4238 heartbeater.cc:511] Master 127.0.116.254:46015 requested a full tablet report, sending...
I20260926 20:20:55.529843 4047 ts_manager.cc:194] Registered new tserver with Master: e3a169a4829a4b769fb2936a778d88bd (127.0.116.193:38439)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.194@KRBTEST.COM" created.
I20260926 20:20:55.531037 4047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:59029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
I20260926 20:20:55.545957 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46015
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:38155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:55.659546 4243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:55.659817 4243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:55.659902 4243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:55.664090 4243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:55.664147 4243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:55.664247 4243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:20:55.669523 4243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:38155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:55.670574 4243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:55.671653 4243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:20:55.681505 4243 server_base.cc:1034] running on GCE node
W20260926 20:20:55.681463 4251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:55.681464 4248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:55.681479 4249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:55.682132 4243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:55.682802 4243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:55.684023 4243 hybrid_clock.cc:648] HybridClock initialized: now 1790454055683987 us; error 47 us; skew 500 ppm
Sep 26 20:20:55 dist-test-slave-qsm9 krb5kdc[4002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:55.686704 4243 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:20:55.687566 4243 webserver.cc:460] Webserver started at http://127.0.116.194:42899/ using document root <none> and password file <none>
I20260926 20:20:55.688077 4243 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:55.688133 4243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:55.688293 4243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:55.690255 4243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "bc579a7951dc4230a070fb3e0e78b1ae"
format_stamp: "Formatted at 2026-09-26 20:20:55 on dist-test-slave-qsm9"
server_key: "e9220272e06aecfe16fe8cc1eefd6122"
server_key_iv: "98607a5a2a400f5a34f5dfb58e0a099e"
server_key_version: "encryptionkey@0"
I20260926 20:20:55.690780 4243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "bc579a7951dc4230a070fb3e0e78b1ae"
format_stamp: "Formatted at 2026-09-26 20:20:55 on dist-test-slave-qsm9"
server_key: "e9220272e06aecfe16fe8cc1eefd6122"
server_key_iv: "98607a5a2a400f5a34f5dfb58e0a099e"
server_key_version: "encryptionkey@0"
I20260926 20:20:55.694836 4243 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:20:55.697269 4258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:55.698482 4243 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 20:20:55.698575 4243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "bc579a7951dc4230a070fb3e0e78b1ae"
format_stamp: "Formatted at 2026-09-26 20:20:55 on dist-test-slave-qsm9"
server_key: "e9220272e06aecfe16fe8cc1eefd6122"
server_key_iv: "98607a5a2a400f5a34f5dfb58e0a099e"
server_key_version: "encryptionkey@0"
I20260926 20:20:55.698660 4243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:55.709357 4243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:55.712883 4243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:55.713053 4243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:55.713608 4243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:20:55.714438 4243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:55.714488 4243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:55.714533 4243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:55.714548 4243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:55.723904 4243 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:46297
I20260926 20:20:55.723930 4371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:46297 every 8 connection(s)
I20260926 20:20:55.724881 4243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:20:55.732781 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 4243
I20260926 20:20:55.732980 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:20:55.733362 467 external_mini_cluster.cc:1428] Setting key c3082858ca40c6d43cd4a6ebc4d74b08
Sep 26 20:20:55 dist-test-slave-qsm9 krb5kdc[4002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:55.740196 4086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:55.726709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:44447 (local address 127.0.116.254:46015)
0926 20:20:55.726862 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:55.726866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:55.727791 (+ 925us) server_negotiation.cc:407] Connection header received
0926 20:20:55.729219 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:55.729223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:55.729284 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:55.729395 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:55.730631 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:55.731170 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:55.732460 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:55.732699 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:55.732815 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:55.735966 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:55.735972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:55.735978 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:55.736015 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:55.737624 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:55.738426 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:55.738429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:55.738431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:55.738522 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:55.738922 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:55.738926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:55.738928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:55.739130 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:55.739250 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:55.739885 (+ 635us) server_negotiation.cc:299] Negotiation successful
0926 20:20:55.740021 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:20:55.741111 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:55.727124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46015 (local address 127.0.116.194:44447)
0926 20:20:55.727633 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:55.727678 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:55.728992 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:55.729596 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:55.729604 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:55.729979 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:55.730481 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:55.730495 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:55.731302 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:55.731305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:55.732310 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:55.732320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:55.732861 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:55.732865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:55.733098 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:55.733820 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:55.733837 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:55.735774 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:55.737753 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:55.737760 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:55.737784 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:55.738247 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:55.738629 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:55.738634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:55.738636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:55.738805 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:55.739248 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:55.739257 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:55.739599 (+ 342us) client_negotiation.cc:769] Sending connection context
0926 20:20:55.739822 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 20:20:55.740195 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":104,"threads_started":1}
I20260926 20:20:55.742404 4372 heartbeater.cc:347] Connected to a master server at 127.0.116.254:46015
I20260926 20:20:55.742682 4372 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:55.743213 4372 heartbeater.cc:511] Master 127.0.116.254:46015 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.116.195@KRBTEST.COM; defaulting to no policy
I20260926 20:20:55.744464 4047 ts_manager.cc:194] Registered new tserver with Master: bc579a7951dc4230a070fb3e0e78b1ae (127.0.116.194:46297)
I20260926 20:20:55.745049 4047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:44447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
I20260926 20:20:55.763437 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46015
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:38155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:55.896333 4377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:55.896615 4377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:55.896660 4377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:55.900889 4377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:55.900956 4377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:55.901055 4377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:20:55.906756 4377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:38155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:55.907976 4377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:55.909035 4377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:55.918781 4382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:55.918984 4377 server_base.cc:1034] running on GCE node
W20260926 20:20:55.918793 4385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:55.918817 4383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:55.919607 4377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:55.920300 4377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:55.921458 4377 hybrid_clock.cc:648] HybridClock initialized: now 1790454055921441 us; error 29 us; skew 500 ppm
Sep 26 20:20:55 dist-test-slave-qsm9 krb5kdc[4002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:55.924230 4377 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:20:55.925164 4377 webserver.cc:460] Webserver started at http://127.0.116.195:43273/ using document root <none> and password file <none>
I20260926 20:20:55.925709 4377 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:55.925770 4377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:55.926023 4377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:55.928131 4377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "4d5603981fc54958a8fc6d09066a479c"
format_stamp: "Formatted at 2026-09-26 20:20:55 on dist-test-slave-qsm9"
server_key: "98e454de83c2ebec285658eb5e0a5d21"
server_key_iv: "edaf6ae54743f867ad4d6fc7f7609cde"
server_key_version: "encryptionkey@0"
I20260926 20:20:55.928709 4377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "4d5603981fc54958a8fc6d09066a479c"
format_stamp: "Formatted at 2026-09-26 20:20:55 on dist-test-slave-qsm9"
server_key: "98e454de83c2ebec285658eb5e0a5d21"
server_key_iv: "edaf6ae54743f867ad4d6fc7f7609cde"
server_key_version: "encryptionkey@0"
I20260926 20:20:55.932446 4377 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:20:55.935760 4392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:55.937354 4377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:20:55.937480 4377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "4d5603981fc54958a8fc6d09066a479c"
format_stamp: "Formatted at 2026-09-26 20:20:55 on dist-test-slave-qsm9"
server_key: "98e454de83c2ebec285658eb5e0a5d21"
server_key_iv: "edaf6ae54743f867ad4d6fc7f7609cde"
server_key_version: "encryptionkey@0"
I20260926 20:20:55.937616 4377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:55.978593 4377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:55.982231 4377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:55.982419 4377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:55.982990 4377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:20:55.983944 4377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:55.983997 4377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:55.984041 4377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:55.984064 4377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:55.993381 4377 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:43283
I20260926 20:20:55.993438 4505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:43283 every 8 connection(s)
I20260926 20:20:55.994426 4377 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:20:56.001034 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 4377
I20260926 20:20:56.001202 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:20:56.001492 467 external_mini_cluster.cc:1428] Setting key b2ce7ef4a9e8c1c6027c72c17420770b
Sep 26 20:20:56 dist-test-slave-qsm9 krb5kdc[4002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:56.009397 4086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:55.996526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:53243 (local address 127.0.116.254:46015)
0926 20:20:55.996692 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:55.996696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:55.997634 (+ 938us) server_negotiation.cc:407] Connection header received
0926 20:20:55.999057 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:55.999061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:55.999123 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:55.999228 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:56.000515 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:56.001054 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:56.002564 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:56.002953 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:56.003077 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:56.005672 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:56.005675 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:56.005678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:56.005703 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:56.007071 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:56.007617 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:56.007620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:56.007621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:56.007700 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:56.008142 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:56.008145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:56.008146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:56.008384 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:56.008478 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:56.009147 (+ 669us) server_negotiation.cc:299] Negotiation successful
0926 20:20:56.009271 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 20:20:56.010361 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:55.996964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46015 (local address 127.0.116.195:53243)
0926 20:20:55.997465 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:55.997510 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:55.998813 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:55.999416 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:55.999424 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:55.999865 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:56.000352 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:56.000366 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:56.001179 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:56.001185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:56.002416 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:56.002428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:56.003128 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:56.003134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:56.003286 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:56.003961 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:56.003978 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:56.005490 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:56.007189 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:56.007194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:56.007207 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:56.007517 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:56.007819 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:56.007822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:56.007823 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:56.008003 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:56.008491 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:56.008498 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:56.008758 (+ 260us) client_negotiation.cc:769] Sending connection context
0926 20:20:56.009065 (+ 307us) client_negotiation.cc:240] Negotiation successful
0926 20:20:56.009411 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":113,"threads_started":1}
I20260926 20:20:56.011634 4506 heartbeater.cc:347] Connected to a master server at 127.0.116.254:46015
I20260926 20:20:56.011875 4506 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:56.012346 4506 heartbeater.cc:511] Master 127.0.116.254:46015 requested a full tablet report, sending...
I20260926 20:20:56.013387 4047 ts_manager.cc:194] Registered new tserver with Master: 4d5603981fc54958a8fc6d09066a479c (127.0.116.195:43283)
I20260926 20:20:56.014011 4047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:53243
I20260926 20:20:56.015671 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:20:56 dist-test-slave-qsm9 krb5kdc[4002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:20:56 dist-test-slave-qsm9 krb5kdc[4002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.193@KRBTEST.COM
I20260926 20:20:56.042105 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:56.032275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55344 (local address 127.0.116.193:38439)
0926 20:20:56.032588 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:56.032594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:56.032619 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:20:56.032716 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:56.032722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:56.032936 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:56.033067 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:56.033759 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:56.034543 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:56.035374 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:56.035814 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:56.035910 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:56.038358 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:56.038376 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:56.038390 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:56.038416 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:56.040212 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:56.040672 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:56.040678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:56.040679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:56.040771 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:56.041023 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:56.041026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:56.041027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:56.041290 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:56.041587 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:56.041737 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 20:20:56.041928 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":105,"threads_started":1}
Sep 26 20:20:56 dist-test-slave-qsm9 krb5kdc[4002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:56.053604 4086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:56.045064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52436 (local address 127.0.116.254:46015)
0926 20:20:56.045185 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:56.045188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:56.045215 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:20:56.045335 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:56.045338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:56.045378 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:56.045450 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:56.046046 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:56.046556 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:56.047540 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:56.047868 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:56.047991 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:56.050496 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:56.050500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:56.050502 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:56.050526 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:56.052036 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:56.052431 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:56.052434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:56.052435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:56.052516 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:56.052744 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:56.052746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:56.052747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:56.052977 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:56.053079 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:56.053309 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 20:20:56.053461 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 20:20:56.054049 4047 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52436
I20260926 20:20:56.054648 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 4109
I20260926 20:20:56.060427 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 4243
I20260926 20:20:56.065572 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 4377
I20260926 20:20:56.071108 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 4015
2026-09-26T20:20:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2201 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:20:56 dist-test-slave-qsm9 krb5kdc[4518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:20:56 dist-test-slave-qsm9 krb5kdc[4518](info): set up 2 sockets
Sep 26 20:20:56 dist-test-slave-qsm9 krb5kdc[4518](info): commencing operation
krb5kdc: starting...
W20260926 20:20:57.140039 467 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:20:57 dist-test-slave-qsm9 krb5kdc[4518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:20:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:20:57.270658 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:43341
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:44183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:43341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:57.384981 4531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:57.385346 4531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:57.385429 4531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:57.389664 4531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:57.389771 4531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:57.389808 4531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:57.389849 4531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:57.389885 4531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:57.396978 4531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:44183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:43341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:43341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:57.398573 4531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:57.399974 4531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:57.410780 4539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:57.410912 4536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:57.410784 4537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:57.411243 4531 server_base.cc:1034] running on GCE node
I20260926 20:20:57.412006 4531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:57.413244 4531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:57.414443 4531 hybrid_clock.cc:648] HybridClock initialized: now 1790454057414410 us; error 47 us; skew 500 ppm
Sep 26 20:20:57 dist-test-slave-qsm9 krb5kdc[4518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:57.423697 4531 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:20:57.424578 4531 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:57.424679 4531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:57.424926 4531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:57.427177 4531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "bf04a4d474734eb2b1e7da07ba79fecd"
format_stamp: "Formatted at 2026-09-26 20:20:57 on dist-test-slave-qsm9"
server_key: "1b21c4b81564df255a36884bb98fb7ea"
server_key_iv: "357e244c1ecff8cf6015d5bbb16b6f06"
server_key_version: "encryptionkey@0"
I20260926 20:20:57.427780 4531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "bf04a4d474734eb2b1e7da07ba79fecd"
format_stamp: "Formatted at 2026-09-26 20:20:57 on dist-test-slave-qsm9"
server_key: "1b21c4b81564df255a36884bb98fb7ea"
server_key_iv: "357e244c1ecff8cf6015d5bbb16b6f06"
server_key_version: "encryptionkey@0"
I20260926 20:20:57.431749 4531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:20:57.434274 4545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:57.435575 4531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:20:57.435735 4531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "bf04a4d474734eb2b1e7da07ba79fecd"
format_stamp: "Formatted at 2026-09-26 20:20:57 on dist-test-slave-qsm9"
server_key: "1b21c4b81564df255a36884bb98fb7ea"
server_key_iv: "357e244c1ecff8cf6015d5bbb16b6f06"
server_key_version: "encryptionkey@0"
I20260926 20:20:57.435856 4531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:57.457773 4531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:57.461514 4531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:57.461742 4531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:57.469451 4531 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:43341
I20260926 20:20:57.469774 4597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:43341 every 8 connection(s)
I20260926 20:20:57.470209 4531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:57.473366 4598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:57.477568 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 4531
I20260926 20:20:57.477739 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:20:57.478062 467 external_mini_cluster.cc:1428] Setting key 310bee923f4ef50f701ca26193a59dc0
I20260926 20:20:57.480306 4598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd: Bootstrap starting.
I20260926 20:20:57.483472 4598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:57.484504 4598 log.cc:824] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:57.487182 4598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd: No bootstrap required, opened a new log
Sep 26 20:20:57 dist-test-slave-qsm9 krb5kdc[4518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:57.491953 4598 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf04a4d474734eb2b1e7da07ba79fecd" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43341 } }
I20260926 20:20:57.492223 4598 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:57.492290 4598 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf04a4d474734eb2b1e7da07ba79fecd, State: Initialized, Role: FOLLOWER
I20260926 20:20:57.492801 4598 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf04a4d474734eb2b1e7da07ba79fecd" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43341 } }
I20260926 20:20:57.492982 4598 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:57.493058 4598 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:57.493155 4598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:57.494211 4598 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf04a4d474734eb2b1e7da07ba79fecd" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43341 } }
I20260926 20:20:57.494645 4598 leader_election.cc:302] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf04a4d474734eb2b1e7da07ba79fecd; no voters:
I20260926 20:20:57.495100 4598 leader_election.cc:288] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:57.495183 4603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:57.495414 4603 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd [term 1 LEADER]: Becoming Leader. State: Replica: bf04a4d474734eb2b1e7da07ba79fecd, State: Running, Role: LEADER
I20260926 20:20:57.495779 4603 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf04a4d474734eb2b1e7da07ba79fecd" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43341 } }
I20260926 20:20:57.496057 4601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:57.479617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54840 (local address 127.0.116.254:43341)
0926 20:20:57.480146 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:57.480169 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:57.480217 (+ 48us) server_negotiation.cc:407] Connection header received
0926 20:20:57.481162 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:57.481189 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:57.481709 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:57.482146 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:57.482856 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:57.484564 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:57.485570 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:57.486305 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:57.486494 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:57.489542 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:57.489550 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:57.489571 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:57.489609 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:57.492487 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:57.493033 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:57.493040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:57.493046 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:57.493178 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:57.493531 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:57.493535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:57.493537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:57.493986 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:57.494200 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:57.494508 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 20:20:57.494886 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":157,"threads_started":1}
I20260926 20:20:57.496608 4598 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:57.497572 4604 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf04a4d474734eb2b1e7da07ba79fecd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf04a4d474734eb2b1e7da07ba79fecd" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43341 } } }
I20260926 20:20:57.497736 4604 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:57.497995 4604 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf04a4d474734eb2b1e7da07ba79fecd. Latest consensus state: current_term: 1 leader_uuid: "bf04a4d474734eb2b1e7da07ba79fecd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf04a4d474734eb2b1e7da07ba79fecd" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43341 } } }
I20260926 20:20:57.498085 4604 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:57.498958 4611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:57.502013 4611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:57.507478 4611 catalog_manager.cc:1347] Generated new cluster ID: 045b228d4e924f4184508d2be042303e
I20260926 20:20:57.507581 4611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:57.533227 4611 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:57.534788 4611 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:57.545967 4611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd: Generated new TSK 0
I20260926 20:20:57.546902 4611 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
I20260926 20:20:57.586750 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:43341
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:44183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:57.703025 4624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:57.703290 4624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:57.703336 4624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:57.707373 4624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:57.707448 4624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:57.707527 4624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:20:57.713065 4624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:44183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:43341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:57.714159 4624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:57.715255 4624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:57.725051 4629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:57.725068 4632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:57.725090 4630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:57.725342 4624 server_base.cc:1034] running on GCE node
I20260926 20:20:57.725875 4624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:57.726549 4624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:57.727742 4624 hybrid_clock.cc:648] HybridClock initialized: now 1790454057727693 us; error 57 us; skew 500 ppm
Sep 26 20:20:57 dist-test-slave-qsm9 krb5kdc[4518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:57.730595 4624 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:20:57.731359 4624 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:57.731431 4624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:57.731680 4624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:57.733819 4624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "1a7e01b42c274182804e17aca3a0683c"
format_stamp: "Formatted at 2026-09-26 20:20:57 on dist-test-slave-qsm9"
server_key: "a8ccadb70e68e4dfba32567851ab9fa3"
server_key_iv: "4d0b389970dcce0225c4e8d7f2c15aa5"
server_key_version: "encryptionkey@0"
I20260926 20:20:57.734356 4624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "1a7e01b42c274182804e17aca3a0683c"
format_stamp: "Formatted at 2026-09-26 20:20:57 on dist-test-slave-qsm9"
server_key: "a8ccadb70e68e4dfba32567851ab9fa3"
server_key_iv: "4d0b389970dcce0225c4e8d7f2c15aa5"
server_key_version: "encryptionkey@0"
I20260926 20:20:57.738309 4624 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:20:57.741065 4638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:57.742421 4624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:20:57.742525 4624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "1a7e01b42c274182804e17aca3a0683c"
format_stamp: "Formatted at 2026-09-26 20:20:57 on dist-test-slave-qsm9"
server_key: "a8ccadb70e68e4dfba32567851ab9fa3"
server_key_iv: "4d0b389970dcce0225c4e8d7f2c15aa5"
server_key_version: "encryptionkey@0"
I20260926 20:20:57.742619 4624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:57.763685 4624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:57.766709 4624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:57.766865 4624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:57.767414 4624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:20:57.768306 4624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:57.768353 4624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:57.768433 4624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:57.768463 4624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:57.777792 4624 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:33175
I20260926 20:20:57.777900 4751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:33175 every 8 connection(s)
I20260926 20:20:57.778438 4624 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:20:57.783746 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 4624
I20260926 20:20:57.783891 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:20:57.784242 467 external_mini_cluster.cc:1428] Setting key 82e6879d2442cef590187c527b81b589
Sep 26 20:20:57 dist-test-slave-qsm9 krb5kdc[4518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:57.793291 4601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:57.780291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:41651 (local address 127.0.116.254:43341)
0926 20:20:57.780441 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:57.780446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:57.781412 (+ 966us) server_negotiation.cc:407] Connection header received
0926 20:20:57.782782 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:57.782785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:57.782832 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:57.782909 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:57.784289 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:57.785001 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:57.786304 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:57.786570 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:57.786686 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:57.789284 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:57.789287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:57.789289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:57.789313 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:57.790810 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:57.791356 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:57.791359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:57.791361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:57.791485 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:57.791965 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:57.791968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:57.791970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:57.792280 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:57.792396 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:57.792996 (+ 600us) server_negotiation.cc:299] Negotiation successful
0926 20:20:57.793125 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.0.116.194@KRBTEST.COM; defaulting to no policy
I20260926 20:20:57.794294 4754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:57.780751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:43341 (local address 127.0.116.193:41651)
0926 20:20:57.781259 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:57.781302 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:57.782556 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:57.783095 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:57.783102 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:57.783469 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:57.784130 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:57.784147 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:57.785138 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:57.785143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:57.786171 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:57.786181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:57.786729 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:57.786733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:57.786911 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:57.787596 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:57.787612 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:57.789105 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:57.790956 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:57.790962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:57.790978 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:57.791243 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:57.791600 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:57.791603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:57.791605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:57.791849 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:57.792441 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:57.792448 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:57.792712 (+ 264us) client_negotiation.cc:769] Sending connection context
0926 20:20:57.792957 (+ 245us) client_negotiation.cc:240] Negotiation successful
0926 20:20:57.793304 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":107,"threads_started":1}
I20260926 20:20:57.795756 4752 heartbeater.cc:347] Connected to a master server at 127.0.116.254:43341
I20260926 20:20:57.796073 4752 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.194@KRBTEST.COM" created.
I20260926 20:20:57.796667 4752 heartbeater.cc:511] Master 127.0.116.254:43341 requested a full tablet report, sending...
I20260926 20:20:57.798537 4558 ts_manager.cc:194] Registered new tserver with Master: 1a7e01b42c274182804e17aca3a0683c (127.0.116.193:33175)
I20260926 20:20:57.799799 4558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:41651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
I20260926 20:20:57.815764 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:43341
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:44183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:57.928159 4757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:57.928436 4757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:57.928483 4757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:57.932569 4757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:57.932626 4757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:57.932721 4757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:20:57.938169 4757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:44183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:43341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:57.939328 4757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:57.940503 4757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:57.950649 4765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:57.950673 4762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:57.950700 4763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:57.951155 4757 server_base.cc:1034] running on GCE node
I20260926 20:20:57.951654 4757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:57.952304 4757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:57.953490 4757 hybrid_clock.cc:648] HybridClock initialized: now 1790454057953472 us; error 28 us; skew 500 ppm
Sep 26 20:20:57 dist-test-slave-qsm9 krb5kdc[4518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:57.956346 4757 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:20:57.957127 4757 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:57.957190 4757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:57.957401 4757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:57.959462 4757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "fc933ac42f264efbae7ec16cdb061f47"
format_stamp: "Formatted at 2026-09-26 20:20:57 on dist-test-slave-qsm9"
server_key: "6ed499a2ac8e60255bd56654a3b596f0"
server_key_iv: "816473d45d6d93070e39942cea4dfb5e"
server_key_version: "encryptionkey@0"
I20260926 20:20:57.960088 4757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "fc933ac42f264efbae7ec16cdb061f47"
format_stamp: "Formatted at 2026-09-26 20:20:57 on dist-test-slave-qsm9"
server_key: "6ed499a2ac8e60255bd56654a3b596f0"
server_key_iv: "816473d45d6d93070e39942cea4dfb5e"
server_key_version: "encryptionkey@0"
I20260926 20:20:57.964020 4757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:20:57.966445 4771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:57.967805 4757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:20:57.967967 4757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "fc933ac42f264efbae7ec16cdb061f47"
format_stamp: "Formatted at 2026-09-26 20:20:57 on dist-test-slave-qsm9"
server_key: "6ed499a2ac8e60255bd56654a3b596f0"
server_key_iv: "816473d45d6d93070e39942cea4dfb5e"
server_key_version: "encryptionkey@0"
I20260926 20:20:57.968086 4757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:57.993261 4757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:57.996587 4757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:57.996801 4757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:57.997426 4757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:20:57.998453 4757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:57.998523 4757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:57.998600 4757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:57.998651 4757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:58.007779 4757 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:37417
I20260926 20:20:58.007797 4884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:37417 every 8 connection(s)
I20260926 20:20:58.008392 4757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:20:58.012318 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 4757
I20260926 20:20:58.012444 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:20:58.012771 467 external_mini_cluster.cc:1428] Setting key 44feb38886a44a0f71ff4c7e899fbcda
Sep 26 20:20:58 dist-test-slave-qsm9 krb5kdc[4518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.116.195@KRBTEST.COM; defaulting to no policy
I20260926 20:20:58.024524 4601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:58.010088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:36217 (local address 127.0.116.254:43341)
0926 20:20:58.010237 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:58.010240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:58.011163 (+ 923us) server_negotiation.cc:407] Connection header received
0926 20:20:58.013075 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:58.013078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:58.013128 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:58.013192 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:58.014606 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.015125 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.016798 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.017172 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.017300 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:58.020319 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:58.020327 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:58.020332 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:58.020369 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:58.022176 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:58.022769 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:58.022775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:58.022779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:58.022875 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:58.023260 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:58.023264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:58.023265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:58.023581 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:58.023665 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:58.024274 (+ 609us) server_negotiation.cc:299] Negotiation successful
0926 20:20:58.024385 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:20:58.025604 4887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:58.010501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:43341 (local address 127.0.116.194:36217)
0926 20:20:58.010986 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:58.011034 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:58.012796 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:58.013410 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:58.013425 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:58.013816 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:58.014431 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:58.014455 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.015257 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.015264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:58.016665 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:58.016675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.017323 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.017326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:58.017498 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:58.018184 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:58.018202 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:58.020119 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:58.022326 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:58.022332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:58.022345 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:58.022641 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:58.022967 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:58.022969 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:58.022971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:58.023107 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:58.023693 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:58.023700 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:58.023985 (+ 285us) client_negotiation.cc:769] Sending connection context
0926 20:20:58.024229 (+ 244us) client_negotiation.cc:240] Negotiation successful
0926 20:20:58.024588 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.195@KRBTEST.COM" created.
I20260926 20:20:58.026990 4885 heartbeater.cc:347] Connected to a master server at 127.0.116.254:43341
I20260926 20:20:58.027256 4885 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:58.027787 4885 heartbeater.cc:511] Master 127.0.116.254:43341 requested a full tablet report, sending...
I20260926 20:20:58.028884 4558 ts_manager.cc:194] Registered new tserver with Master: fc933ac42f264efbae7ec16cdb061f47 (127.0.116.194:37417)
I20260926 20:20:58.029778 4558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:36217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
I20260926 20:20:58.043881 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:43341
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:44183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:20:58.156276 4890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:58.156548 4890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:58.156594 4890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:58.160687 4890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:58.160748 4890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:58.160810 4890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:20:58.166215 4890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:44183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:43341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:58.167263 4890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:58.168242 4890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:58.178115 4895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:58.178212 4890 server_base.cc:1034] running on GCE node
W20260926 20:20:58.178109 4898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:58.178109 4896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:58.178917 4890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:58.179653 4890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:58.180837 4890 hybrid_clock.cc:648] HybridClock initialized: now 1790454058180813 us; error 44 us; skew 500 ppm
Sep 26 20:20:58 dist-test-slave-qsm9 krb5kdc[4518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:20:58.183975 4890 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:20:58.184715 4890 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:58.184777 4890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:58.184994 4890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:58.187142 4890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "07aedb8ebf3f4971bb69a81a752918cf"
format_stamp: "Formatted at 2026-09-26 20:20:58 on dist-test-slave-qsm9"
server_key: "8faf5412a400ecc622e966ea33cc8ae9"
server_key_iv: "1a654c667c5ae7a307853d132b769221"
server_key_version: "encryptionkey@0"
I20260926 20:20:58.187775 4890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "07aedb8ebf3f4971bb69a81a752918cf"
format_stamp: "Formatted at 2026-09-26 20:20:58 on dist-test-slave-qsm9"
server_key: "8faf5412a400ecc622e966ea33cc8ae9"
server_key_iv: "1a654c667c5ae7a307853d132b769221"
server_key_version: "encryptionkey@0"
I20260926 20:20:58.191687 4890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 20:20:58.194336 4904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:58.195685 4890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:20:58.195814 4890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "07aedb8ebf3f4971bb69a81a752918cf"
format_stamp: "Formatted at 2026-09-26 20:20:58 on dist-test-slave-qsm9"
server_key: "8faf5412a400ecc622e966ea33cc8ae9"
server_key_iv: "1a654c667c5ae7a307853d132b769221"
server_key_version: "encryptionkey@0"
I20260926 20:20:58.195935 4890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:58.230598 4890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:58.234125 4890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:58.234356 4890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:58.234992 4890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:20:58.235903 4890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:58.235975 4890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:58.236054 4890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:58.236104 4890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:58.245213 4890 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:35389
I20260926 20:20:58.245229 5017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:35389 every 8 connection(s)
I20260926 20:20:58.245862 4890 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:20:58.251103 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 4890
I20260926 20:20:58.251245 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:20:58.251622 467 external_mini_cluster.cc:1428] Setting key a5857e388e2ac6ec08c34cc019e6a0c3
Sep 26 20:20:58 dist-test-slave-qsm9 krb5kdc[4518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:20:58.260053 4601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:58.247817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:49903 (local address 127.0.116.254:43341)
0926 20:20:58.247990 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:58.247995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:58.248842 (+ 847us) server_negotiation.cc:407] Connection header received
0926 20:20:58.250170 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:58.250173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:58.250220 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:58.250295 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:58.251613 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.252322 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.253732 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.254032 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.254122 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:58.256542 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:58.256545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:58.256547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:58.256573 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:58.258037 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:58.258543 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:58.258546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:58.258547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:58.258626 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:58.258940 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:58.258943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:58.258944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:58.259122 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:58.259198 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:58.259794 (+ 596us) server_negotiation.cc:299] Negotiation successful
0926 20:20:58.259927 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 20:20:58.260959 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:58.248253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:43341 (local address 127.0.116.195:49903)
0926 20:20:58.248685 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:58.248727 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:58.249978 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:58.250459 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:58.250466 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:58.250812 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:58.251437 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:58.251453 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.252467 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.252473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:58.253558 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:58.253594 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.254167 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.254170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:58.254311 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:58.254895 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:20:58.254912 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:58.256373 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:58.258159 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:58.258163 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:58.258177 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:58.258455 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:58.258704 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:20:58.258706 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:20:58.258708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:20:58.258839 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:20:58.259245 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:58.259251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:58.259565 (+ 314us) client_negotiation.cc:769] Sending connection context
0926 20:20:58.259741 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 20:20:58.260051 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
I20260926 20:20:58.262254 5018 heartbeater.cc:347] Connected to a master server at 127.0.116.254:43341
I20260926 20:20:58.262490 5018 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:58.262907 5018 heartbeater.cc:511] Master 127.0.116.254:43341 requested a full tablet report, sending...
I20260926 20:20:58.263978 4558 ts_manager.cc:194] Registered new tserver with Master: 07aedb8ebf3f4971bb69a81a752918cf (127.0.116.195:35389)
I20260926 20:20:58.264752 4558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:49903
I20260926 20:20:58.265683 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:20:58.275044 4601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:58.267657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54850 (local address 127.0.116.254:43341)
0926 20:20:58.267793 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:58.267798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:58.267976 (+ 178us) server_negotiation.cc:407] Connection header received
0926 20:20:58.268073 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:58.268075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:58.268113 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:58.268178 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:58.268835 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.269353 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.270118 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.270321 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.270397 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:58.271937 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:58.271940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:58.271942 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:20:58.271963 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:58.273259 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:58.273829 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:58.273832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:58.273833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:58.273910 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:20:58.274145 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:20:58.274147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:20:58.274148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:20:58.274379 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:58.274480 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:58.274713 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 20:20:58.274842 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:20:58.277922 4558 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:20:58.280934 4558 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:20:58.295544 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:58.289428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48912 (local address 127.0.116.194:37417)
0926 20:20:58.290474 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:58.290488 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:58.290511 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:20:58.290601 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:58.290607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:58.290828 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:58.291044 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:58.291348 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.292226 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.293888 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.294708 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.295052 (+ 344us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:58.295193 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:58.295244 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 20:20:58.295342 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":914,"thread_start_us":122,"threads_started":1}
I20260926 20:20:58.295552 5029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:58.289278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:37417 (local address 127.0.0.1:48912)
0926 20:20:58.289649 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:58.289705 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:58.289844 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:58.291042 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:58.291047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:58.291069 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:58.291222 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:58.291229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.292361 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.292366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:58.293765 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:58.293774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.294848 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.294851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:58.294976 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:58.295088 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 20:20:58.295192 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 20:20:58.295346 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":107,"thread_start_us":83,"threads_started":1}
I20260926 20:20:58.295580 5033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:58.290152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41992 (local address 127.0.116.193:33175)
0926 20:20:58.290599 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:58.290606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:58.290629 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:20:58.290706 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:58.290711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:58.290893 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:58.291058 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:58.291413 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.292602 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.294050 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.294816 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.294934 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:58.295077 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:58.295313 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 20:20:58.295406 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":87,"threads_started":1}
I20260926 20:20:58.295573 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:58.289668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:33175 (local address 127.0.0.1:41992)
0926 20:20:58.290328 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:58.290342 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:58.290439 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:58.291199 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:58.291202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:58.291215 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:58.291296 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:58.291304 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.292766 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.292770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:58.293915 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:58.293927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.295077 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.295080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:58.295163 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:58.295185 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:20:58.295277 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 20:20:58.295376 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":3,"thread_start_us":97,"threads_started":1}
I20260926 20:20:58.296175 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:58.290043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:35389 (local address 127.0.0.1:40434)
0926 20:20:58.290619 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:58.290631 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:58.290702 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:58.291445 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:58.291449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:58.291466 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:58.291589 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:58.291595 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.292897 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.292901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:58.293732 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:58.293743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.294900 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.294906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:58.294991 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:58.295940 (+ 949us) client_negotiation.cc:769] Sending connection context
0926 20:20:58.296014 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 20:20:58.296086 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":296,"thread_start_us":82,"threads_started":1}
I20260926 20:20:58.296808 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:58.290313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40434 (local address 127.0.116.195:35389)
0926 20:20:58.290854 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:58.290864 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:58.290886 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:20:58.290957 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:58.290963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:58.291127 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:58.291254 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:58.291708 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.292774 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.293877 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.294708 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.294823 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:58.294971 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:58.296550 (+ 1579us) server_negotiation.cc:299] Negotiation successful
0926 20:20:58.296665 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":119,"threads_started":1}
I20260926 20:20:58.298892 4819 tablet_service.cc:1467] Processing CreateTablet for tablet 407367b5efad4d68891c8956649fa435 (DEFAULT_TABLE table=test-table [id=74a82ae4baff4941a92e7d13bca6dd22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:58.298897 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 407367b5efad4d68891c8956649fa435 (DEFAULT_TABLE table=test-table [id=74a82ae4baff4941a92e7d13bca6dd22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:58.299986 4819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407367b5efad4d68891c8956649fa435. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:58.299616 4952 tablet_service.cc:1467] Processing CreateTablet for tablet 407367b5efad4d68891c8956649fa435 (DEFAULT_TABLE table=test-table [id=74a82ae4baff4941a92e7d13bca6dd22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:58.299996 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407367b5efad4d68891c8956649fa435. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:58.300544 4952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407367b5efad4d68891c8956649fa435. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:58.305996 5035 tablet_bootstrap.cc:492] T 407367b5efad4d68891c8956649fa435 P fc933ac42f264efbae7ec16cdb061f47: Bootstrap starting.
I20260926 20:20:58.307512 5036 tablet_bootstrap.cc:492] T 407367b5efad4d68891c8956649fa435 P 1a7e01b42c274182804e17aca3a0683c: Bootstrap starting.
I20260926 20:20:58.307866 5037 tablet_bootstrap.cc:492] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf: Bootstrap starting.
I20260926 20:20:58.308633 5035 tablet_bootstrap.cc:654] T 407367b5efad4d68891c8956649fa435 P fc933ac42f264efbae7ec16cdb061f47: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:58.309566 5036 tablet_bootstrap.cc:654] T 407367b5efad4d68891c8956649fa435 P 1a7e01b42c274182804e17aca3a0683c: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:58.309973 5037 tablet_bootstrap.cc:654] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:58.309993 5035 log.cc:824] T 407367b5efad4d68891c8956649fa435 P fc933ac42f264efbae7ec16cdb061f47: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:58.310305 5036 log.cc:824] T 407367b5efad4d68891c8956649fa435 P 1a7e01b42c274182804e17aca3a0683c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:58.311213 5037 log.cc:824] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:58.312144 5035 tablet_bootstrap.cc:492] T 407367b5efad4d68891c8956649fa435 P fc933ac42f264efbae7ec16cdb061f47: No bootstrap required, opened a new log
I20260926 20:20:58.312227 5036 tablet_bootstrap.cc:492] T 407367b5efad4d68891c8956649fa435 P 1a7e01b42c274182804e17aca3a0683c: No bootstrap required, opened a new log
I20260926 20:20:58.312379 5035 ts_tablet_manager.cc:1397] T 407367b5efad4d68891c8956649fa435 P fc933ac42f264efbae7ec16cdb061f47: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 20:20:58.312403 5036 ts_tablet_manager.cc:1397] T 407367b5efad4d68891c8956649fa435 P 1a7e01b42c274182804e17aca3a0683c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:20:58.312984 5037 tablet_bootstrap.cc:492] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf: No bootstrap required, opened a new log
I20260926 20:20:58.313144 5037 ts_tablet_manager.cc:1397] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260926 20:20:58.315681 5035 raft_consensus.cc:348] T 407367b5efad4d68891c8956649fa435 P fc933ac42f264efbae7ec16cdb061f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc933ac42f264efbae7ec16cdb061f47" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37417 } } peers { permanent_uuid: "07aedb8ebf3f4971bb69a81a752918cf" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35389 } } peers { permanent_uuid: "1a7e01b42c274182804e17aca3a0683c" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33175 } }
I20260926 20:20:58.315943 5035 raft_consensus.cc:374] T 407367b5efad4d68891c8956649fa435 P fc933ac42f264efbae7ec16cdb061f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:58.315996 5035 raft_consensus.cc:729] T 407367b5efad4d68891c8956649fa435 P fc933ac42f264efbae7ec16cdb061f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc933ac42f264efbae7ec16cdb061f47, State: Initialized, Role: FOLLOWER
I20260926 20:20:58.316414 5035 consensus_queue.cc:260] T 407367b5efad4d68891c8956649fa435 P fc933ac42f264efbae7ec16cdb061f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc933ac42f264efbae7ec16cdb061f47" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37417 } } peers { permanent_uuid: "07aedb8ebf3f4971bb69a81a752918cf" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35389 } } peers { permanent_uuid: "1a7e01b42c274182804e17aca3a0683c" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33175 } }
I20260926 20:20:58.316819 5037 raft_consensus.cc:348] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc933ac42f264efbae7ec16cdb061f47" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37417 } } peers { permanent_uuid: "07aedb8ebf3f4971bb69a81a752918cf" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35389 } } peers { permanent_uuid: "1a7e01b42c274182804e17aca3a0683c" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33175 } }
I20260926 20:20:58.317044 4885 heartbeater.cc:503] Master 127.0.116.254:43341 was elected leader, sending a full tablet report...
I20260926 20:20:58.317061 5037 raft_consensus.cc:374] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:58.317109 5037 raft_consensus.cc:729] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07aedb8ebf3f4971bb69a81a752918cf, State: Initialized, Role: FOLLOWER
I20260926 20:20:58.317441 5035 ts_tablet_manager.cc:1428] T 407367b5efad4d68891c8956649fa435 P fc933ac42f264efbae7ec16cdb061f47: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 20:20:58.317493 5037 consensus_queue.cc:260] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc933ac42f264efbae7ec16cdb061f47" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37417 } } peers { permanent_uuid: "07aedb8ebf3f4971bb69a81a752918cf" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35389 } } peers { permanent_uuid: "1a7e01b42c274182804e17aca3a0683c" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33175 } }
I20260926 20:20:58.318107 5018 heartbeater.cc:503] Master 127.0.116.254:43341 was elected leader, sending a full tablet report...
I20260926 20:20:58.316781 5036 raft_consensus.cc:348] T 407367b5efad4d68891c8956649fa435 P 1a7e01b42c274182804e17aca3a0683c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc933ac42f264efbae7ec16cdb061f47" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37417 } } peers { permanent_uuid: "07aedb8ebf3f4971bb69a81a752918cf" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35389 } } peers { permanent_uuid: "1a7e01b42c274182804e17aca3a0683c" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33175 } }
I20260926 20:20:58.318235 5036 raft_consensus.cc:374] T 407367b5efad4d68891c8956649fa435 P 1a7e01b42c274182804e17aca3a0683c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:58.318288 5036 raft_consensus.cc:729] T 407367b5efad4d68891c8956649fa435 P 1a7e01b42c274182804e17aca3a0683c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a7e01b42c274182804e17aca3a0683c, State: Initialized, Role: FOLLOWER
I20260926 20:20:58.318385 5037 ts_tablet_manager.cc:1428] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20260926 20:20:58.318660 5036 consensus_queue.cc:260] T 407367b5efad4d68891c8956649fa435 P 1a7e01b42c274182804e17aca3a0683c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc933ac42f264efbae7ec16cdb061f47" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37417 } } peers { permanent_uuid: "07aedb8ebf3f4971bb69a81a752918cf" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35389 } } peers { permanent_uuid: "1a7e01b42c274182804e17aca3a0683c" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33175 } }
I20260926 20:20:58.319216 4752 heartbeater.cc:503] Master 127.0.116.254:43341 was elected leader, sending a full tablet report...
I20260926 20:20:58.319480 5036 ts_tablet_manager.cc:1428] T 407367b5efad4d68891c8956649fa435 P 1a7e01b42c274182804e17aca3a0683c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260926 20:20:58.496826 5019 tablet.cc:2369] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:20:58.509462 4886 tablet.cc:2369] T 407367b5efad4d68891c8956649fa435 P fc933ac42f264efbae7ec16cdb061f47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:20:58.518366 5042 raft_consensus.cc:482] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:20:58.518621 5042 raft_consensus.cc:504] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc933ac42f264efbae7ec16cdb061f47" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37417 } } peers { permanent_uuid: "07aedb8ebf3f4971bb69a81a752918cf" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35389 } } peers { permanent_uuid: "1a7e01b42c274182804e17aca3a0683c" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33175 } }
I20260926 20:20:58.519974 5042 leader_election.cc:288] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc933ac42f264efbae7ec16cdb061f47 (127.0.116.194:37417), 1a7e01b42c274182804e17aca3a0683c (127.0.116.193:33175)
I20260926 20:20:58.524312 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:58.520304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:33175 (local address 127.0.116.195:50279)
0926 20:20:58.520403 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:58.520442 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:58.520547 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:58.520935 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:58.520938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:58.520958 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:58.521088 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:58.521095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.521864 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.521870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:58.523069 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:58.523080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.523895 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.523898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:58.523994 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:58.524014 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 20:20:58.524084 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 20:20:58.524138 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":4}
I20260926 20:20:58.524336 5033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:58.520534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:50279 (local address 127.0.116.193:33175)
0926 20:20:58.520684 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:58.520687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:58.520702 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:20:58.520755 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:58.520757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:58.520801 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:58.520907 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:58.521222 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.521737 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.523201 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.523715 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.523829 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:58.523880 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:58.524121 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 20:20:58.524206 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:20:58.524710 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:58.520448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:48001 (local address 127.0.116.194:37417)
0926 20:20:58.520601 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:58.520606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:58.521079 (+ 473us) server_negotiation.cc:407] Connection header received
0926 20:20:58.521219 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:58.521223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:58.521269 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:58.521347 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:20:58.521613 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.522144 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.523565 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.524104 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.524216 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:58.524260 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:58.524490 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 20:20:58.524585 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 20:20:58.525122 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:58.520304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:37417 (local address 127.0.116.195:48001)
0926 20:20:58.520977 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:58.520989 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:58.521102 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:58.521366 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:58.521369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:58.521384 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:20:58.521503 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:58.521509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.522317 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.522321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:58.523420 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:20:58.523428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.524222 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.524225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:20:58.524324 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:58.524357 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 20:20:58.524893 (+ 536us) client_negotiation.cc:240] Negotiation successful
0926 20:20:58.524961 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":357,"thread_start_us":139,"threads_started":1}
I20260926 20:20:58.525079 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407367b5efad4d68891c8956649fa435" candidate_uuid: "07aedb8ebf3f4971bb69a81a752918cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a7e01b42c274182804e17aca3a0683c" is_pre_election: true
I20260926 20:20:58.525425 4706 raft_consensus.cc:2454] T 407367b5efad4d68891c8956649fa435 P 1a7e01b42c274182804e17aca3a0683c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07aedb8ebf3f4971bb69a81a752918cf in term 0.
I20260926 20:20:58.525786 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407367b5efad4d68891c8956649fa435" candidate_uuid: "07aedb8ebf3f4971bb69a81a752918cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc933ac42f264efbae7ec16cdb061f47" is_pre_election: true
I20260926 20:20:58.525931 4906 leader_election.cc:302] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07aedb8ebf3f4971bb69a81a752918cf, 1a7e01b42c274182804e17aca3a0683c; no voters:
I20260926 20:20:58.526099 4839 raft_consensus.cc:2454] T 407367b5efad4d68891c8956649fa435 P fc933ac42f264efbae7ec16cdb061f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07aedb8ebf3f4971bb69a81a752918cf in term 0.
I20260926 20:20:58.526264 5042 raft_consensus.cc:2781] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:20:58.526362 5042 raft_consensus.cc:482] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:20:58.526429 5042 raft_consensus.cc:3037] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:58.527551 5042 raft_consensus.cc:504] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc933ac42f264efbae7ec16cdb061f47" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37417 } } peers { permanent_uuid: "07aedb8ebf3f4971bb69a81a752918cf" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35389 } } peers { permanent_uuid: "1a7e01b42c274182804e17aca3a0683c" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33175 } }
I20260926 20:20:58.527904 5042 leader_election.cc:288] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf [CANDIDATE]: Term 1 election: Requested vote from peers fc933ac42f264efbae7ec16cdb061f47 (127.0.116.194:37417), 1a7e01b42c274182804e17aca3a0683c (127.0.116.193:33175)
I20260926 20:20:58.528260 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407367b5efad4d68891c8956649fa435" candidate_uuid: "07aedb8ebf3f4971bb69a81a752918cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc933ac42f264efbae7ec16cdb061f47"
I20260926 20:20:58.528398 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407367b5efad4d68891c8956649fa435" candidate_uuid: "07aedb8ebf3f4971bb69a81a752918cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a7e01b42c274182804e17aca3a0683c"
I20260926 20:20:58.528407 4839 raft_consensus.cc:3037] T 407367b5efad4d68891c8956649fa435 P fc933ac42f264efbae7ec16cdb061f47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:58.528553 4706 raft_consensus.cc:3037] T 407367b5efad4d68891c8956649fa435 P 1a7e01b42c274182804e17aca3a0683c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:58.529498 4839 raft_consensus.cc:2454] T 407367b5efad4d68891c8956649fa435 P fc933ac42f264efbae7ec16cdb061f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07aedb8ebf3f4971bb69a81a752918cf in term 1.
W20260926 20:20:58.529668 4753 tablet.cc:2369] T 407367b5efad4d68891c8956649fa435 P 1a7e01b42c274182804e17aca3a0683c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:20:58.529716 4706 raft_consensus.cc:2454] T 407367b5efad4d68891c8956649fa435 P 1a7e01b42c274182804e17aca3a0683c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07aedb8ebf3f4971bb69a81a752918cf in term 1.
I20260926 20:20:58.529794 4907 leader_election.cc:302] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07aedb8ebf3f4971bb69a81a752918cf, fc933ac42f264efbae7ec16cdb061f47; no voters:
I20260926 20:20:58.530112 5042 raft_consensus.cc:2781] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:58.530337 5042 raft_consensus.cc:686] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf [term 1 LEADER]: Becoming Leader. State: Replica: 07aedb8ebf3f4971bb69a81a752918cf, State: Running, Role: LEADER
I20260926 20:20:58.530666 5042 consensus_queue.cc:237] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc933ac42f264efbae7ec16cdb061f47" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37417 } } peers { permanent_uuid: "07aedb8ebf3f4971bb69a81a752918cf" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35389 } } peers { permanent_uuid: "1a7e01b42c274182804e17aca3a0683c" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33175 } }
I20260926 20:20:58.533812 4558 catalog_manager.cc:5495] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 07aedb8ebf3f4971bb69a81a752918cf (127.0.116.195). New cstate: current_term: 1 leader_uuid: "07aedb8ebf3f4971bb69a81a752918cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc933ac42f264efbae7ec16cdb061f47" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07aedb8ebf3f4971bb69a81a752918cf" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a7e01b42c274182804e17aca3a0683c" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33175 } health_report { overall_health: UNKNOWN } } }
I20260926 20:20:58.591598 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:58.587687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40436 (local address 127.0.116.195:35389)
0926 20:20:58.587851 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:58.587858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:58.587953 (+ 95us) server_negotiation.cc:407] Connection header received
0926 20:20:58.588165 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:58.588168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:58.588233 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:58.588348 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:20:58.588625 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.589171 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.590045 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:20:58.590275 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:20:58.590392 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:20:58.590710 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:20:58.591135 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:20:58.591268 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:58.591314 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 20:20:58.591379 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 20:20:58.597692 4706 raft_consensus.cc:1261] T 407367b5efad4d68891c8956649fa435 P 1a7e01b42c274182804e17aca3a0683c [term 1 FOLLOWER]: Refusing update from remote peer 07aedb8ebf3f4971bb69a81a752918cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:58.597692 4839 raft_consensus.cc:1261] T 407367b5efad4d68891c8956649fa435 P fc933ac42f264efbae7ec16cdb061f47 [term 1 FOLLOWER]: Refusing update from remote peer 07aedb8ebf3f4971bb69a81a752918cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:58.598377 5042 consensus_queue.cc:1035] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc933ac42f264efbae7ec16cdb061f47" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:20:58.598634 5045 consensus_queue.cc:1035] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a7e01b42c274182804e17aca3a0683c" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:20:58.606120 5049 mvcc.cc:205] Tried to move back new op lower bound from 7333699824008511488 to 7333699823744126976. Current Snapshot: MvccSnapshot[applied={T|T < 7333699824008511488}]
I20260926 20:20:58.606948 5050 mvcc.cc:205] Tried to move back new op lower bound from 7333699824008511488 to 7333699823744126976. Current Snapshot: MvccSnapshot[applied={T|T < 7333699824008511488}]
I20260926 20:20:58.613875 4558 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:20:58.614073 4558 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:20:58.616582 4558 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf04a4d474734eb2b1e7da07ba79fecd: Sending DeleteTablet for 3 replicas of tablet 407367b5efad4d68891c8956649fa435
I20260926 20:20:58.617403 4686 tablet_service.cc:1514] Processing DeleteTablet for tablet 407367b5efad4d68891c8956649fa435 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:58 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:41992
I20260926 20:20:58.617408 4952 tablet_service.cc:1514] Processing DeleteTablet for tablet 407367b5efad4d68891c8956649fa435 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:58 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:40434
I20260926 20:20:58.617403 4819 tablet_service.cc:1514] Processing DeleteTablet for tablet 407367b5efad4d68891c8956649fa435 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:58 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:48912
I20260926 20:20:58.617920 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 4624
I20260926 20:20:58.617962 5059 tablet_replica.cc:330] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf: stopping tablet replica
I20260926 20:20:58.618233 5059 raft_consensus.cc:2229] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:20:58.618541 5059 raft_consensus.cc:2258] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:20:58.618916 5057 tablet_replica.cc:330] T 407367b5efad4d68891c8956649fa435 P fc933ac42f264efbae7ec16cdb061f47: stopping tablet replica
I20260926 20:20:58.619203 5057 raft_consensus.cc:2229] T 407367b5efad4d68891c8956649fa435 P fc933ac42f264efbae7ec16cdb061f47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:20:58.619539 5057 raft_consensus.cc:2258] T 407367b5efad4d68891c8956649fa435 P fc933ac42f264efbae7ec16cdb061f47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:20:58.621076 5059 ts_tablet_manager.cc:1905] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:20:58.623801 5059 ts_tablet_manager.cc:1918] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:20:58.623911 5059 log.cc:1196] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/407367b5efad4d68891c8956649fa435
I20260926 20:20:58.624229 5059 ts_tablet_manager.cc:1939] T 407367b5efad4d68891c8956649fa435 P 07aedb8ebf3f4971bb69a81a752918cf: Deleting consensus metadata
W20260926 20:20:58.625236 4547 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.116.193:33175 (error 108)
I20260926 20:20:58.625618 5057 ts_tablet_manager.cc:1905] T 407367b5efad4d68891c8956649fa435 P fc933ac42f264efbae7ec16cdb061f47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:20:58.625764 4548 catalog_manager.cc:4841] TS 07aedb8ebf3f4971bb69a81a752918cf (127.0.116.195:35389): tablet 407367b5efad4d68891c8956649fa435 (table test-table [id=74a82ae4baff4941a92e7d13bca6dd22]) successfully deleted
I20260926 20:20:58.626327 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 4757
I20260926 20:20:58.626442 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:58.626099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:33175 (local address 127.0.0.1:42002)
0926 20:20:58.626278 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:58.626347 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.193:33175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260926 20:20:58.627192 4547 catalog_manager.cc:4584] TS 1a7e01b42c274182804e17aca3a0683c (127.0.116.193:33175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 407367b5efad4d68891c8956649fa435: Network error: Client connection negotiation failed: client connection to 127.0.116.193:33175: connect: Connection refused (error 111)
I20260926 20:20:58.632257 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 4890
I20260926 20:20:58.633469 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:58.632497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:37417 (local address 127.0.0.1:48926)
0926 20:20:58.633294 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:58.633362 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.194:37417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":692}
I20260926 20:20:58.637223 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 4531
2026-09-26T20:20:58Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2564 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T20:20:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:58Z Disabled control of system clock
I20260926 20:20:58.666800 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:34511
--webserver_interface=127.0.116.254
--webserver_port=0
--builtin_ntp_servers=127.0.116.212:33873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:34511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 20:20:58.778316 5064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:58.778604 5064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:58.778642 5064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:58.782766 5064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:20:58.782832 5064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:58.782848 5064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:58.782866 5064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:20:58.782881 5064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:20:58.787772 5064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:33873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:34511
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:34511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:58.788993 5064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:58.790395 5064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:58.798753 5069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:58.798956 5064 server_base.cc:1034] running on GCE node
W20260926 20:20:58.798921 5072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:58.798761 5070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:58.799714 5064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:58.800751 5064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:58.801934 5064 hybrid_clock.cc:648] HybridClock initialized: now 1790454058801921 us; error 27 us; skew 500 ppm
I20260926 20:20:58.803805 5064 webserver.cc:460] Webserver started at http://127.0.116.254:34973/ using document root <none> and password file <none>
I20260926 20:20:58.804340 5064 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:58.804399 5064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:58.804563 5064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:58.806562 5064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "1d29f66f6ddc40d18a65948585100359"
format_stamp: "Formatted at 2026-09-26 20:20:58 on dist-test-slave-qsm9"
server_key: "49fe4ee7d96b23d1839e8da4dace9896"
server_key_iv: "4aa72e6a640a39ad618e0d5d401fc343"
server_key_version: "encryptionkey@0"
I20260926 20:20:58.807085 5064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "1d29f66f6ddc40d18a65948585100359"
format_stamp: "Formatted at 2026-09-26 20:20:58 on dist-test-slave-qsm9"
server_key: "49fe4ee7d96b23d1839e8da4dace9896"
server_key_iv: "4aa72e6a640a39ad618e0d5d401fc343"
server_key_version: "encryptionkey@0"
I20260926 20:20:58.810791 5064 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 20:20:58.813344 5078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:58.814615 5064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:20:58.814720 5064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "1d29f66f6ddc40d18a65948585100359"
format_stamp: "Formatted at 2026-09-26 20:20:58 on dist-test-slave-qsm9"
server_key: "49fe4ee7d96b23d1839e8da4dace9896"
server_key_iv: "4aa72e6a640a39ad618e0d5d401fc343"
server_key_version: "encryptionkey@0"
I20260926 20:20:58.814810 5064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:58.819172 5064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:58.819710 5064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:58.819839 5064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:58.826500 5064 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:34511
I20260926 20:20:58.826504 5130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:34511 every 8 connection(s)
I20260926 20:20:58.827409 5064 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:20:58.830585 5131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:58.832719 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 5064
I20260926 20:20:58.832856 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:20:58.833132 467 external_mini_cluster.cc:1428] Setting key 63d464cdf34109fba9b4a78ef0e4b2bc
I20260926 20:20:58.836675 5131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359: Bootstrap starting.
I20260926 20:20:58.838778 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:58.834647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41166 (local address 127.0.116.254:34511)
0926 20:20:58.835339 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:58.835353 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:58.835387 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:20:58.836059 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:58.836077 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:58.836430 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:58.836762 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:58.837560 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:58.837564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:58.837576 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:58.837586 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:58.837642 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:58.837657 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:58.837804 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:58.837875 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 20:20:58.838122 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":150,"threads_started":1}
I20260926 20:20:58.839115 5131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:58.839967 5131 log.cc:824] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:58.841866 5131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359: No bootstrap required, opened a new log
I20260926 20:20:58.845729 5131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d29f66f6ddc40d18a65948585100359" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34511 } }
I20260926 20:20:58.845960 5131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:58.845995 5131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d29f66f6ddc40d18a65948585100359, State: Initialized, Role: FOLLOWER
I20260926 20:20:58.846444 5131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d29f66f6ddc40d18a65948585100359" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34511 } }
I20260926 20:20:58.846621 5131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:20:58.846694 5131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:20:58.846798 5131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:58.847899 5131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d29f66f6ddc40d18a65948585100359" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34511 } }
I20260926 20:20:58.848197 5131 leader_election.cc:302] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d29f66f6ddc40d18a65948585100359; no voters:
I20260926 20:20:58.848466 5131 leader_election.cc:288] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:20:58.848605 5136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:58.848865 5136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359 [term 1 LEADER]: Becoming Leader. State: Replica: 1d29f66f6ddc40d18a65948585100359, State: Running, Role: LEADER
I20260926 20:20:58.849200 5136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d29f66f6ddc40d18a65948585100359" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34511 } }
I20260926 20:20:58.849752 5131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:20:58.850874 5137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d29f66f6ddc40d18a65948585100359" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d29f66f6ddc40d18a65948585100359" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34511 } } }
I20260926 20:20:58.851004 5137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:58.851311 5138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d29f66f6ddc40d18a65948585100359. Latest consensus state: current_term: 1 leader_uuid: "1d29f66f6ddc40d18a65948585100359" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d29f66f6ddc40d18a65948585100359" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34511 } } }
I20260926 20:20:58.851419 5138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359 [sys.catalog]: This master's current role is: LEADER
I20260926 20:20:58.851634 5145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:20:58.854290 5145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:20:58.859756 5145 catalog_manager.cc:1347] Generated new cluster ID: 5d7b88a4f65b4f25b29d2fa3ff6ab6c8
I20260926 20:20:58.859822 5145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:20:58.871464 5145 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:20:58.871826 5145 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:20:58.878611 5145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359: Generated new TSK 0
I20260926 20:20:58.879346 5145 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:20:58.887310 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:34511
--builtin_ntp_servers=127.0.116.212:33873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 20:20:59.002439 5155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:59.002812 5155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:59.002858 5155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:59.006927 5155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:59.006990 5155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:59.007050 5155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:20:59.011890 5155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:33873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:59.013103 5155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:59.014564 5155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:59.024861 5163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:59.024843 5161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:59.024974 5155 server_base.cc:1034] running on GCE node
W20260926 20:20:59.024888 5160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:59.025648 5155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:59.026469 5155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:59.027693 5155 hybrid_clock.cc:648] HybridClock initialized: now 1790454059027660 us; error 57 us; skew 500 ppm
I20260926 20:20:59.029738 5155 webserver.cc:460] Webserver started at http://127.0.116.193:33243/ using document root <none> and password file <none>
I20260926 20:20:59.030405 5155 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:59.030507 5155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:59.030766 5155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:59.033147 5155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "da8773618d6b4dc6998e1032b9d93793"
format_stamp: "Formatted at 2026-09-26 20:20:59 on dist-test-slave-qsm9"
server_key: "4aa2354bd2f384d548fe21c259f96425"
server_key_iv: "3acb8e6f9af2b3adab4f09dc7f3dab3a"
server_key_version: "encryptionkey@0"
I20260926 20:20:59.033787 5155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "da8773618d6b4dc6998e1032b9d93793"
format_stamp: "Formatted at 2026-09-26 20:20:59 on dist-test-slave-qsm9"
server_key: "4aa2354bd2f384d548fe21c259f96425"
server_key_iv: "3acb8e6f9af2b3adab4f09dc7f3dab3a"
server_key_version: "encryptionkey@0"
I20260926 20:20:59.037997 5155 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:20:59.040756 5169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:59.042126 5155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:20:59.042287 5155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "da8773618d6b4dc6998e1032b9d93793"
format_stamp: "Formatted at 2026-09-26 20:20:59 on dist-test-slave-qsm9"
server_key: "4aa2354bd2f384d548fe21c259f96425"
server_key_iv: "3acb8e6f9af2b3adab4f09dc7f3dab3a"
server_key_version: "encryptionkey@0"
I20260926 20:20:59.042431 5155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:59.047313 5155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:59.048002 5155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:59.048199 5155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:59.048759 5155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:20:59.049695 5155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:59.049779 5155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:59.049877 5155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:59.049933 5155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:59.059765 5155 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:35801
I20260926 20:20:59.059818 5282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:35801 every 8 connection(s)
I20260926 20:20:59.060757 5155 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:20:59.063829 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 5155
I20260926 20:20:59.064015 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:20:59.064385 467 external_mini_cluster.cc:1428] Setting key 60881f61f8d9aeff62d40be873d34e0f
I20260926 20:20:59.066473 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:34511
--builtin_ntp_servers=127.0.116.212:33873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 20:20:59.068027 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:59.063014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:38377 (local address 127.0.116.254:34511)
0926 20:20:59.063199 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:59.063203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:59.064498 (+ 1295us) server_negotiation.cc:407] Connection header received
0926 20:20:59.065964 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:59.065966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:59.066003 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:59.066103 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:59.067557 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:59.067560 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:59.067561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:59.067573 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:59.067592 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:59.067600 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:59.067673 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:59.067851 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 20:20:59.067889 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 20:20:59.069041 5285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:59.063396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:34511 (local address 127.0.116.193:38377)
0926 20:20:59.064308 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:59.064368 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:59.065703 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:59.066337 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:59.066349 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:59.066475 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:59.067183 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:20:59.067188 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:59.067250 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:20:59.067350 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:59.067689 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:59.067701 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:59.067709 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 20:20:59.067906 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 20:20:59.068169 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":118,"threads_started":1}
I20260926 20:20:59.070216 5283 heartbeater.cc:347] Connected to a master server at 127.0.116.254:34511
I20260926 20:20:59.070468 5283 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:59.071015 5283 heartbeater.cc:511] Master 127.0.116.254:34511 requested a full tablet report, sending...
I20260926 20:20:59.072662 5095 ts_manager.cc:194] Registered new tserver with Master: da8773618d6b4dc6998e1032b9d93793 (127.0.116.193:35801)
W20260926 20:20:59.187353 5286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:59.187713 5286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:59.187793 5286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:59.192018 5286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:59.192083 5286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:59.192174 5286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:20:59.197235 5286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:33873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:59.198351 5286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:59.199834 5286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:59.209628 5291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:59.209628 5292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:59.209623 5294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:59.209679 5286 server_base.cc:1034] running on GCE node
I20260926 20:20:59.210273 5286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:59.210994 5286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:59.212167 5286 hybrid_clock.cc:648] HybridClock initialized: now 1790454059212130 us; error 47 us; skew 500 ppm
I20260926 20:20:59.214241 5286 webserver.cc:460] Webserver started at http://127.0.116.194:39673/ using document root <none> and password file <none>
I20260926 20:20:59.214840 5286 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:59.214903 5286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:59.215113 5286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:59.217249 5286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "a1d960dcc9dc49d38c24cfe0461956e5"
format_stamp: "Formatted at 2026-09-26 20:20:59 on dist-test-slave-qsm9"
server_key: "987080d44b7ccd43a611e397a0b31656"
server_key_iv: "0609f62061758b61699f3e9078b9ecf1"
server_key_version: "encryptionkey@0"
I20260926 20:20:59.217821 5286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "a1d960dcc9dc49d38c24cfe0461956e5"
format_stamp: "Formatted at 2026-09-26 20:20:59 on dist-test-slave-qsm9"
server_key: "987080d44b7ccd43a611e397a0b31656"
server_key_iv: "0609f62061758b61699f3e9078b9ecf1"
server_key_version: "encryptionkey@0"
I20260926 20:20:59.222273 5286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:20:59.224982 5300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:59.226217 5286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:20:59.226362 5286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "a1d960dcc9dc49d38c24cfe0461956e5"
format_stamp: "Formatted at 2026-09-26 20:20:59 on dist-test-slave-qsm9"
server_key: "987080d44b7ccd43a611e397a0b31656"
server_key_iv: "0609f62061758b61699f3e9078b9ecf1"
server_key_version: "encryptionkey@0"
I20260926 20:20:59.226473 5286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:59.231137 5286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:59.231827 5286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:59.232004 5286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:59.232545 5286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:20:59.233397 5286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:59.233439 5286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:59.233476 5286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:59.233531 5286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:59.242156 5286 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:41495
I20260926 20:20:59.242184 5413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:41495 every 8 connection(s)
I20260926 20:20:59.243079 5286 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:20:59.244057 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 5286
I20260926 20:20:59.244190 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:20:59.244480 467 external_mini_cluster.cc:1428] Setting key b25aaafe6156e7698c3bc9bd8a993c7c
I20260926 20:20:59.246893 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:34511
--builtin_ntp_servers=127.0.116.212:33873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 20:20:59.250773 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:59.245235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:50845 (local address 127.0.116.254:34511)
0926 20:20:59.245423 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:59.245428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:59.246835 (+ 1407us) server_negotiation.cc:407] Connection header received
0926 20:20:59.248817 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:59.248820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:59.248868 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:59.248973 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:59.250224 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:59.250228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:59.250230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:59.250250 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:59.250270 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:59.250279 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:59.250387 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:59.250542 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 20:20:59.250608 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260926 20:20:59.251857 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:59.245792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:34511 (local address 127.0.116.194:50845)
0926 20:20:59.246663 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:59.246723 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:59.248576 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:59.249154 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:59.249161 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:59.249255 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:59.249901 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:20:59.249902 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:59.249948 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:20:59.250020 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:59.250369 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:59.250377 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:59.250382 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:20:59.250579 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 20:20:59.250887 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":136,"threads_started":1}
I20260926 20:20:59.253206 5414 heartbeater.cc:347] Connected to a master server at 127.0.116.254:34511
I20260926 20:20:59.253494 5414 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:59.253928 5414 heartbeater.cc:511] Master 127.0.116.254:34511 requested a full tablet report, sending...
I20260926 20:20:59.255121 5095 ts_manager.cc:194] Registered new tserver with Master: a1d960dcc9dc49d38c24cfe0461956e5 (127.0.116.194:41495)
W20260926 20:20:59.359787 5417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:20:59.360095 5417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:20:59.360163 5417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:20:59.364276 5417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:20:59.364353 5417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:20:59.364441 5417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:20:59.369495 5417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:33873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:20:59.370601 5417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:20:59.372040 5417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:20:59.381860 5422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:20:59.381841 5423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:59.381917 5417 server_base.cc:1034] running on GCE node
W20260926 20:20:59.381852 5425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:20:59.382494 5417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:20:59.383172 5417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:20:59.384344 5417 hybrid_clock.cc:648] HybridClock initialized: now 1790454059384332 us; error 28 us; skew 500 ppm
I20260926 20:20:59.386211 5417 webserver.cc:460] Webserver started at http://127.0.116.195:43563/ using document root <none> and password file <none>
I20260926 20:20:59.386746 5417 fs_manager.cc:359] Metadata directory not provided
I20260926 20:20:59.386811 5417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:20:59.386973 5417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:20:59.389019 5417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "a0462c8c5b324c8d99d211131bbb1f07"
format_stamp: "Formatted at 2026-09-26 20:20:59 on dist-test-slave-qsm9"
server_key: "97a657b24b2afa198446738ffc90d64b"
server_key_iv: "34b1cab1aa259b46f02bbf6da30edd83"
server_key_version: "encryptionkey@0"
I20260926 20:20:59.389520 5417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "a0462c8c5b324c8d99d211131bbb1f07"
format_stamp: "Formatted at 2026-09-26 20:20:59 on dist-test-slave-qsm9"
server_key: "97a657b24b2afa198446738ffc90d64b"
server_key_iv: "34b1cab1aa259b46f02bbf6da30edd83"
server_key_version: "encryptionkey@0"
I20260926 20:20:59.393666 5417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:20:59.397163 5431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:59.398833 5417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:20:59.399003 5417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "a0462c8c5b324c8d99d211131bbb1f07"
format_stamp: "Formatted at 2026-09-26 20:20:59 on dist-test-slave-qsm9"
server_key: "97a657b24b2afa198446738ffc90d64b"
server_key_iv: "34b1cab1aa259b46f02bbf6da30edd83"
server_key_version: "encryptionkey@0"
I20260926 20:20:59.399123 5417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:20:59.403658 5417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:20:59.404258 5417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:20:59.404451 5417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:20:59.404989 5417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:20:59.406036 5417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:20:59.406106 5417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:59.406178 5417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:20:59.406231 5417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:20:59.415015 5417 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:32981
I20260926 20:20:59.415033 5544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:32981 every 8 connection(s)
I20260926 20:20:59.416013 5417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:20:59.422152 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:59.417817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:43847 (local address 127.0.116.254:34511)
0926 20:20:59.417961 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:59.417965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:59.419176 (+ 1211us) server_negotiation.cc:407] Connection header received
0926 20:20:59.420495 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:59.420498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:59.420536 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:59.420611 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:59.421740 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:59.421743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:59.421745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:59.421757 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:59.421776 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:59.421784 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:59.421860 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:59.422011 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 20:20:59.422048 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:20:59.422956 5547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:59.418287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:34511 (local address 127.0.116.195:43847)
0926 20:20:59.419036 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:59.419078 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:59.420302 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:59.420765 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:59.420772 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:59.420860 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:59.421485 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:20:59.421487 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:59.421526 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:20:59.421591 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:59.421856 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:59.421862 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:59.421866 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:20:59.422024 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 20:20:59.422208 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":109,"threads_started":1}
I20260926 20:20:59.423888 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 5417
I20260926 20:20:59.424010 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:20:59.424207 5545 heartbeater.cc:347] Connected to a master server at 127.0.116.254:34511
I20260926 20:20:59.424335 467 external_mini_cluster.cc:1428] Setting key bd8c7d986100d033ae6c59a5d6bafc61
I20260926 20:20:59.424468 5545 heartbeater.cc:464] Registering TS with master...
I20260926 20:20:59.424952 5545 heartbeater.cc:511] Master 127.0.116.254:34511 requested a full tablet report, sending...
I20260926 20:20:59.426023 5094 ts_manager.cc:194] Registered new tserver with Master: a0462c8c5b324c8d99d211131bbb1f07 (127.0.116.195:32981)
I20260926 20:20:59.438381 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:20:59.442243 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:59.440450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41168 (local address 127.0.116.254:34511)
0926 20:20:59.440609 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:59.440613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:59.440827 (+ 214us) server_negotiation.cc:407] Connection header received
0926 20:20:59.440891 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:59.440920 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:59.440957 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:59.441038 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:59.441805 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:59.441808 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:59.441810 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:59.441820 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:59.441839 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:59.441847 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:59.441962 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:59.442031 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 20:20:59.442066 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:20:59.445276 5094 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:20:59.448475 5094 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:20:59.460263 5559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:59.457339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:41495 (local address 127.0.0.1:41254)
0926 20:20:59.458253 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:59.458265 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:59.458330 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:59.458882 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:59.458885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:59.458902 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:59.459332 (+ 430us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:20:59.459336 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:59.459359 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:20:59.459431 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:59.459892 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:59.459901 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:59.459906 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:20:59.460024 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 20:20:59.460093 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"thread_start_us":92,"threads_started":1}
I20260926 20:20:59.460728 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:59.457754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41254 (local address 127.0.116.194:41495)
0926 20:20:59.458087 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:59.458097 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:59.458489 (+ 392us) server_negotiation.cc:407] Connection header received
0926 20:20:59.458552 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:59.458560 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:59.458734 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:59.458882 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:59.459651 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:59.459660 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:59.459673 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:59.459684 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:59.459721 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:59.459749 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:59.459851 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:59.460452 (+ 601us) server_negotiation.cc:299] Negotiation successful
0926 20:20:59.460533 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":99,"threads_started":1}
I20260926 20:20:59.460955 5558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:59.457108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59024 (local address 127.0.116.193:35801)
0926 20:20:59.458054 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:59.458071 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:59.458099 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:20:59.458162 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:59.458167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:59.458373 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:59.459863 (+ 1490us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:59.460411 (+ 548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:59.460418 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:59.460429 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:59.460439 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:59.460477 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:59.460501 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:59.460619 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:59.460734 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 20:20:59.460804 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":837,"thread_start_us":128,"threads_started":1}
I20260926 20:20:59.461382 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:59.456969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:35801 (local address 127.0.0.1:59024)
0926 20:20:59.457668 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:59.457718 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:59.457798 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:59.458605 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:59.458610 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:59.458691 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:59.459684 (+ 993us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:20:59.459686 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:59.459697 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:20:59.459723 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:59.460615 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:59.460619 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:59.460622 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:20:59.461234 (+ 612us) client_negotiation.cc:240] Negotiation successful
0926 20:20:59.461268 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":88,"threads_started":1}
I20260926 20:20:59.461803 5556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:59.457131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:32981 (local address 127.0.0.1:37780)
0926 20:20:59.458177 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:59.458193 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:59.458261 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:59.459564 (+ 1303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:59.459567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:59.459581 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:59.460033 (+ 452us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:20:59.460035 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:59.460044 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:20:59.460076 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:59.461552 (+ 1476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:59.461558 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:59.461563 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:20:59.461655 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 20:20:59.461691 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"thread_start_us":109,"threads_started":1}
I20260926 20:20:59.462278 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:59.457754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37780 (local address 127.0.116.195:32981)
0926 20:20:59.458594 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:59.458606 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:59.458631 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:20:59.458685 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:59.458691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:59.458870 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:59.459005 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:59.460192 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:59.460198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:59.460215 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:59.460224 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:59.460255 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:59.460272 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:59.460385 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:59.462057 (+ 1672us) server_negotiation.cc:299] Negotiation successful
0926 20:20:59.462123 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":159,"threads_started":1}
I20260926 20:20:59.463799 5348 tablet_service.cc:1467] Processing CreateTablet for tablet ac5d69dbc0d64d8a93534399f7a3fcf5 (DEFAULT_TABLE table=test-table [id=8d206d1d583c4e67aa8971e6a8cb2c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:59.464107 5217 tablet_service.cc:1467] Processing CreateTablet for tablet ac5d69dbc0d64d8a93534399f7a3fcf5 (DEFAULT_TABLE table=test-table [id=8d206d1d583c4e67aa8971e6a8cb2c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:59.465109 5348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac5d69dbc0d64d8a93534399f7a3fcf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:59.465181 5217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac5d69dbc0d64d8a93534399f7a3fcf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:59.464905 5479 tablet_service.cc:1467] Processing CreateTablet for tablet ac5d69dbc0d64d8a93534399f7a3fcf5 (DEFAULT_TABLE table=test-table [id=8d206d1d583c4e67aa8971e6a8cb2c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:20:59.465795 5479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac5d69dbc0d64d8a93534399f7a3fcf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:20:59.471305 5562 tablet_bootstrap.cc:492] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5: Bootstrap starting.
I20260926 20:20:59.472798 5563 tablet_bootstrap.cc:492] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a0462c8c5b324c8d99d211131bbb1f07: Bootstrap starting.
I20260926 20:20:59.472862 5564 tablet_bootstrap.cc:492] T ac5d69dbc0d64d8a93534399f7a3fcf5 P da8773618d6b4dc6998e1032b9d93793: Bootstrap starting.
I20260926 20:20:59.474131 5562 tablet_bootstrap.cc:654] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:59.474895 5562 log.cc:824] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:59.474960 5564 tablet_bootstrap.cc:654] T ac5d69dbc0d64d8a93534399f7a3fcf5 P da8773618d6b4dc6998e1032b9d93793: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:59.474960 5563 tablet_bootstrap.cc:654] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a0462c8c5b324c8d99d211131bbb1f07: Neither blocks nor log segments found. Creating new log.
I20260926 20:20:59.475782 5563 log.cc:824] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a0462c8c5b324c8d99d211131bbb1f07: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:59.475782 5564 log.cc:824] T ac5d69dbc0d64d8a93534399f7a3fcf5 P da8773618d6b4dc6998e1032b9d93793: Log is configured to *not* fsync() on all Append() calls
I20260926 20:20:59.476950 5562 tablet_bootstrap.cc:492] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5: No bootstrap required, opened a new log
I20260926 20:20:59.477164 5562 ts_tablet_manager.cc:1397] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:20:59.477706 5563 tablet_bootstrap.cc:492] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a0462c8c5b324c8d99d211131bbb1f07: No bootstrap required, opened a new log
I20260926 20:20:59.477706 5564 tablet_bootstrap.cc:492] T ac5d69dbc0d64d8a93534399f7a3fcf5 P da8773618d6b4dc6998e1032b9d93793: No bootstrap required, opened a new log
I20260926 20:20:59.477895 5563 ts_tablet_manager.cc:1397] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a0462c8c5b324c8d99d211131bbb1f07: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:20:59.477895 5564 ts_tablet_manager.cc:1397] T ac5d69dbc0d64d8a93534399f7a3fcf5 P da8773618d6b4dc6998e1032b9d93793: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 20:20:59.481081 5562 raft_consensus.cc:348] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8773618d6b4dc6998e1032b9d93793" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35801 } } peers { permanent_uuid: "a0462c8c5b324c8d99d211131bbb1f07" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 32981 } } peers { permanent_uuid: "a1d960dcc9dc49d38c24cfe0461956e5" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41495 } }
I20260926 20:20:59.481340 5562 raft_consensus.cc:374] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:59.481393 5562 raft_consensus.cc:729] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1d960dcc9dc49d38c24cfe0461956e5, State: Initialized, Role: FOLLOWER
I20260926 20:20:59.481863 5562 consensus_queue.cc:260] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8773618d6b4dc6998e1032b9d93793" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35801 } } peers { permanent_uuid: "a0462c8c5b324c8d99d211131bbb1f07" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 32981 } } peers { permanent_uuid: "a1d960dcc9dc49d38c24cfe0461956e5" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41495 } }
I20260926 20:20:59.482513 5414 heartbeater.cc:503] Master 127.0.116.254:34511 was elected leader, sending a full tablet report...
I20260926 20:20:59.482477 5564 raft_consensus.cc:348] T ac5d69dbc0d64d8a93534399f7a3fcf5 P da8773618d6b4dc6998e1032b9d93793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8773618d6b4dc6998e1032b9d93793" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35801 } } peers { permanent_uuid: "a0462c8c5b324c8d99d211131bbb1f07" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 32981 } } peers { permanent_uuid: "a1d960dcc9dc49d38c24cfe0461956e5" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41495 } }
I20260926 20:20:59.482488 5563 raft_consensus.cc:348] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a0462c8c5b324c8d99d211131bbb1f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8773618d6b4dc6998e1032b9d93793" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35801 } } peers { permanent_uuid: "a0462c8c5b324c8d99d211131bbb1f07" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 32981 } } peers { permanent_uuid: "a1d960dcc9dc49d38c24cfe0461956e5" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41495 } }
I20260926 20:20:59.482729 5563 raft_consensus.cc:374] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a0462c8c5b324c8d99d211131bbb1f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:59.482729 5564 raft_consensus.cc:374] T ac5d69dbc0d64d8a93534399f7a3fcf5 P da8773618d6b4dc6998e1032b9d93793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:20:59.482785 5564 raft_consensus.cc:729] T ac5d69dbc0d64d8a93534399f7a3fcf5 P da8773618d6b4dc6998e1032b9d93793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da8773618d6b4dc6998e1032b9d93793, State: Initialized, Role: FOLLOWER
I20260926 20:20:59.482785 5563 raft_consensus.cc:729] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a0462c8c5b324c8d99d211131bbb1f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0462c8c5b324c8d99d211131bbb1f07, State: Initialized, Role: FOLLOWER
I20260926 20:20:59.483218 5563 consensus_queue.cc:260] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a0462c8c5b324c8d99d211131bbb1f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8773618d6b4dc6998e1032b9d93793" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35801 } } peers { permanent_uuid: "a0462c8c5b324c8d99d211131bbb1f07" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 32981 } } peers { permanent_uuid: "a1d960dcc9dc49d38c24cfe0461956e5" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41495 } }
I20260926 20:20:59.483218 5564 consensus_queue.cc:260] T ac5d69dbc0d64d8a93534399f7a3fcf5 P da8773618d6b4dc6998e1032b9d93793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8773618d6b4dc6998e1032b9d93793" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35801 } } peers { permanent_uuid: "a0462c8c5b324c8d99d211131bbb1f07" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 32981 } } peers { permanent_uuid: "a1d960dcc9dc49d38c24cfe0461956e5" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41495 } }
I20260926 20:20:59.484043 5562 ts_tablet_manager.cc:1428] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 20:20:59.484180 5564 ts_tablet_manager.cc:1428] T ac5d69dbc0d64d8a93534399f7a3fcf5 P da8773618d6b4dc6998e1032b9d93793: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260926 20:20:59.484179 5563 ts_tablet_manager.cc:1428] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a0462c8c5b324c8d99d211131bbb1f07: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260926 20:20:59.484496 5283 heartbeater.cc:503] Master 127.0.116.254:34511 was elected leader, sending a full tablet report...
I20260926 20:20:59.484496 5545 heartbeater.cc:503] Master 127.0.116.254:34511 was elected leader, sending a full tablet report...
W20260926 20:20:59.494119 5415 tablet.cc:2369] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:20:59.562216 5284 tablet.cc:2369] T ac5d69dbc0d64d8a93534399f7a3fcf5 P da8773618d6b4dc6998e1032b9d93793: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:20:59.666973 5546 tablet.cc:2369] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a0462c8c5b324c8d99d211131bbb1f07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:20:59.743873 5568 raft_consensus.cc:482] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:20:59.744091 5568 raft_consensus.cc:504] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8773618d6b4dc6998e1032b9d93793" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35801 } } peers { permanent_uuid: "a0462c8c5b324c8d99d211131bbb1f07" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 32981 } } peers { permanent_uuid: "a1d960dcc9dc49d38c24cfe0461956e5" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41495 } }
I20260926 20:20:59.745482 5568 leader_election.cc:288] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da8773618d6b4dc6998e1032b9d93793 (127.0.116.193:35801), a0462c8c5b324c8d99d211131bbb1f07 (127.0.116.195:32981)
I20260926 20:20:59.747602 5558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:59.745982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:60351 (local address 127.0.116.193:35801)
0926 20:20:59.746120 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:59.746130 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:59.746145 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:20:59.746187 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:59.746189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:59.746220 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:59.746329 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:59.747117 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:59.747121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:59.747123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:59.747136 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:59.747162 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:59.747175 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:59.747260 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:59.747408 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 20:20:59.747447 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:20:59.747602 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:59.745705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:35801 (local address 127.0.116.194:60351)
0926 20:20:59.745840 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:59.745859 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:59.745950 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:59.746347 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:59.746350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:59.746365 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:59.746934 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:20:59.746937 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:59.746951 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:20:59.746984 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:59.747304 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:59.747306 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:59.747309 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:20:59.747397 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 20:20:59.747458 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 20:20:59.747687 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:59.745930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:43603 (local address 127.0.116.195:32981)
0926 20:20:59.746066 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:59.746071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:59.746282 (+ 211us) server_negotiation.cc:407] Connection header received
0926 20:20:59.746400 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:59.746410 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:59.746444 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:59.746551 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:59.747228 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:59.747232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:59.747234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:59.747247 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:59.747265 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:59.747275 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:59.747334 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:59.747525 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 20:20:59.747567 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:20:59.748025 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:59.745834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:32981 (local address 127.0.116.194:43603)
0926 20:20:59.746192 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:59.746202 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:20:59.746296 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:20:59.746568 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:20:59.746571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:20:59.746585 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:20:59.747057 (+ 472us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:20:59.747059 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:20:59.747069 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:20:59.747091 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:20:59.747403 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:20:59.747406 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:20:59.747409 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:20:59.747846 (+ 437us) client_negotiation.cc:240] Negotiation successful
0926 20:20:59.747884 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":135,"threads_started":1}
I20260926 20:20:59.748258 5236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5d69dbc0d64d8a93534399f7a3fcf5" candidate_uuid: "a1d960dcc9dc49d38c24cfe0461956e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da8773618d6b4dc6998e1032b9d93793" is_pre_election: true
I20260926 20:20:59.748654 5236 raft_consensus.cc:2454] T ac5d69dbc0d64d8a93534399f7a3fcf5 P da8773618d6b4dc6998e1032b9d93793 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1d960dcc9dc49d38c24cfe0461956e5 in term 0.
I20260926 20:20:59.748795 5499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5d69dbc0d64d8a93534399f7a3fcf5" candidate_uuid: "a1d960dcc9dc49d38c24cfe0461956e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0462c8c5b324c8d99d211131bbb1f07" is_pre_election: true
I20260926 20:20:59.749078 5499 raft_consensus.cc:2454] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a0462c8c5b324c8d99d211131bbb1f07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1d960dcc9dc49d38c24cfe0461956e5 in term 0.
I20260926 20:20:59.749155 5301 leader_election.cc:302] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1d960dcc9dc49d38c24cfe0461956e5, da8773618d6b4dc6998e1032b9d93793; no voters:
I20260926 20:20:59.749452 5568 raft_consensus.cc:2781] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:20:59.749552 5568 raft_consensus.cc:482] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:20:59.749595 5568 raft_consensus.cc:3037] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:59.750767 5568 raft_consensus.cc:504] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8773618d6b4dc6998e1032b9d93793" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35801 } } peers { permanent_uuid: "a0462c8c5b324c8d99d211131bbb1f07" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 32981 } } peers { permanent_uuid: "a1d960dcc9dc49d38c24cfe0461956e5" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41495 } }
I20260926 20:20:59.751143 5568 leader_election.cc:288] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5 [CANDIDATE]: Term 1 election: Requested vote from peers da8773618d6b4dc6998e1032b9d93793 (127.0.116.193:35801), a0462c8c5b324c8d99d211131bbb1f07 (127.0.116.195:32981)
I20260926 20:20:59.751479 5236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5d69dbc0d64d8a93534399f7a3fcf5" candidate_uuid: "a1d960dcc9dc49d38c24cfe0461956e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da8773618d6b4dc6998e1032b9d93793"
I20260926 20:20:59.751576 5499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5d69dbc0d64d8a93534399f7a3fcf5" candidate_uuid: "a1d960dcc9dc49d38c24cfe0461956e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0462c8c5b324c8d99d211131bbb1f07"
I20260926 20:20:59.751628 5236 raft_consensus.cc:3037] T ac5d69dbc0d64d8a93534399f7a3fcf5 P da8773618d6b4dc6998e1032b9d93793 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:59.751690 5499 raft_consensus.cc:3037] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a0462c8c5b324c8d99d211131bbb1f07 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:20:59.752976 5499 raft_consensus.cc:2454] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a0462c8c5b324c8d99d211131bbb1f07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1d960dcc9dc49d38c24cfe0461956e5 in term 1.
I20260926 20:20:59.752977 5236 raft_consensus.cc:2454] T ac5d69dbc0d64d8a93534399f7a3fcf5 P da8773618d6b4dc6998e1032b9d93793 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1d960dcc9dc49d38c24cfe0461956e5 in term 1.
I20260926 20:20:59.753284 5302 leader_election.cc:302] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0462c8c5b324c8d99d211131bbb1f07, a1d960dcc9dc49d38c24cfe0461956e5; no voters:
I20260926 20:20:59.753460 5568 raft_consensus.cc:2781] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:20:59.753722 5568 raft_consensus.cc:686] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5 [term 1 LEADER]: Becoming Leader. State: Replica: a1d960dcc9dc49d38c24cfe0461956e5, State: Running, Role: LEADER
I20260926 20:20:59.754076 5568 consensus_queue.cc:237] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8773618d6b4dc6998e1032b9d93793" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35801 } } peers { permanent_uuid: "a0462c8c5b324c8d99d211131bbb1f07" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 32981 } } peers { permanent_uuid: "a1d960dcc9dc49d38c24cfe0461956e5" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41495 } }
I20260926 20:20:59.757530 5092 catalog_manager.cc:5495] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1d960dcc9dc49d38c24cfe0461956e5 (127.0.116.194). New cstate: current_term: 1 leader_uuid: "a1d960dcc9dc49d38c24cfe0461956e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8773618d6b4dc6998e1032b9d93793" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0462c8c5b324c8d99d211131bbb1f07" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 32981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1d960dcc9dc49d38c24cfe0461956e5" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41495 } health_report { overall_health: HEALTHY } } }
I20260926 20:20:59.825161 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:59.823484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41264 (local address 127.0.116.194:41495)
0926 20:20:59.823665 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 20:20:59.823669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:20:59.823685 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:20:59.823790 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:20:59.823792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:20:59.823851 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:20:59.823946 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:20:59.824629 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:20:59.824633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:20:59.824636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:20:59.824654 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:20:59.824673 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:20:59.824681 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:20:59.824780 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 20:20:59.824964 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 20:20:59.825006 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:20:59.831691 5236 raft_consensus.cc:1261] T ac5d69dbc0d64d8a93534399f7a3fcf5 P da8773618d6b4dc6998e1032b9d93793 [term 1 FOLLOWER]: Refusing update from remote peer a1d960dcc9dc49d38c24cfe0461956e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:59.831836 5499 raft_consensus.cc:1261] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a0462c8c5b324c8d99d211131bbb1f07 [term 1 FOLLOWER]: Refusing update from remote peer a1d960dcc9dc49d38c24cfe0461956e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:20:59.832542 5568 consensus_queue.cc:1035] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da8773618d6b4dc6998e1032b9d93793" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:20:59.832758 5572 consensus_queue.cc:1035] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0462c8c5b324c8d99d211131bbb1f07" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 32981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:20:59.847704 5577 mvcc.cc:205] Tried to move back new op lower bound from 7333699829062717440 to 7333699828755550208. Current Snapshot: MvccSnapshot[applied={T|T < 7333699829062717440}]
I20260926 20:20:59.850570 5095 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:20:59.850752 5095 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:20:59.853540 5095 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d29f66f6ddc40d18a65948585100359: Sending DeleteTablet for 3 replicas of tablet ac5d69dbc0d64d8a93534399f7a3fcf5
I20260926 20:20:59.854424 5217 tablet_service.cc:1514] Processing DeleteTablet for tablet ac5d69dbc0d64d8a93534399f7a3fcf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:59 UTC) from {username='slave'} at 127.0.0.1:59024
I20260926 20:20:59.854988 5585 tablet_replica.cc:330] T ac5d69dbc0d64d8a93534399f7a3fcf5 P da8773618d6b4dc6998e1032b9d93793: stopping tablet replica
I20260926 20:20:59.854993 5479 tablet_service.cc:1514] Processing DeleteTablet for tablet ac5d69dbc0d64d8a93534399f7a3fcf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:59 UTC) from {username='slave'} at 127.0.0.1:37780
I20260926 20:20:59.855149 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 5155
I20260926 20:20:59.855476 5348 tablet_service.cc:1514] Processing DeleteTablet for tablet ac5d69dbc0d64d8a93534399f7a3fcf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:20:59 UTC) from {username='slave'} at 127.0.0.1:41254
I20260926 20:20:59.856175 5587 tablet_replica.cc:330] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5: stopping tablet replica
I20260926 20:20:59.856696 5587 raft_consensus.cc:2229] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:20:59.857097 5587 raft_consensus.cc:2258] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:20:59.857535 5586 tablet_replica.cc:330] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a0462c8c5b324c8d99d211131bbb1f07: stopping tablet replica
I20260926 20:20:59.857831 5586 raft_consensus.cc:2229] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a0462c8c5b324c8d99d211131bbb1f07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:20:59.858152 5586 raft_consensus.cc:2258] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a0462c8c5b324c8d99d211131bbb1f07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:20:59.859750 5586 ts_tablet_manager.cc:1905] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a0462c8c5b324c8d99d211131bbb1f07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:20:59.859750 5587 ts_tablet_manager.cc:1905] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:20:59.862703 5586 ts_tablet_manager.cc:1918] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a0462c8c5b324c8d99d211131bbb1f07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:20:59.862717 5587 ts_tablet_manager.cc:1918] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:20:59.862810 5587 log.cc:1196] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/ac5d69dbc0d64d8a93534399f7a3fcf5
I20260926 20:20:59.862948 5586 log.cc:1196] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a0462c8c5b324c8d99d211131bbb1f07: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/ac5d69dbc0d64d8a93534399f7a3fcf5
I20260926 20:20:59.863155 5587 ts_tablet_manager.cc:1939] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a1d960dcc9dc49d38c24cfe0461956e5: Deleting consensus metadata
I20260926 20:20:59.863413 5586 ts_tablet_manager.cc:1939] T ac5d69dbc0d64d8a93534399f7a3fcf5 P a0462c8c5b324c8d99d211131bbb1f07: Deleting consensus metadata
I20260926 20:20:59.864588 5080 catalog_manager.cc:4841] TS a0462c8c5b324c8d99d211131bbb1f07 (127.0.116.195:32981): tablet ac5d69dbc0d64d8a93534399f7a3fcf5 (table test-table [id=8d206d1d583c4e67aa8971e6a8cb2c9a]) successfully deleted
I20260926 20:20:59.864715 5082 catalog_manager.cc:4841] TS a1d960dcc9dc49d38c24cfe0461956e5 (127.0.116.194:41495): tablet ac5d69dbc0d64d8a93534399f7a3fcf5 (table test-table [id=8d206d1d583c4e67aa8971e6a8cb2c9a]) successfully deleted
W20260926 20:20:59.864770 5079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.116.193:35801 (error 108)
I20260926 20:20:59.864987 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 5286
I20260926 20:20:59.866734 5556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:20:59.866135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:35801 (local address 127.0.0.1:59034)
0926 20:20:59.866505 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0926 20:20:59.866619 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.193:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249}
W20260926 20:20:59.867017 5079 catalog_manager.cc:4584] TS da8773618d6b4dc6998e1032b9d93793 (127.0.116.193:35801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ac5d69dbc0d64d8a93534399f7a3fcf5: Network error: Client connection negotiation failed: client connection to 127.0.116.193:35801: connect: Connection refused (error 111)
I20260926 20:20:59.873530 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 5417
I20260926 20:20:59.878880 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 5064
2026-09-26T20:20:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1242 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 20:20:59.901139 467 mini_oidc.cc:152] Starting JWKS server
I20260926 20:20:59.902710 467 webserver.cc:460] Webserver started at https://127.0.0.1:43575/ using document root <none> and password file <none>
I20260926 20:20:59.902849 467 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 20:20:59.903080 467 webserver.cc:460] Webserver started at http://127.0.0.1:45795/ using document root <none> and password file <none>
2026-09-26T20:20:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:20:59Z Disabled control of system clock
I20260926 20:20:59.912187 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:35513
--webserver_interface=127.0.116.254
--webserver_port=0
--builtin_ntp_servers=127.0.116.212:41251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:35513
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43575/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790454033974695-467-0/testchainca.cert with env {}
W20260926 20:21:00.026214 5594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:00.026515 5594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:00.026574 5594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:00.030831 5594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:21:00.030915 5594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:00.030930 5594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:00.030948 5594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:21:00.030963 5594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:21:00.030980 5594 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:21:00.030993 5594 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43575/jwks.json
I20260926 20:21:00.036584 5594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:41251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:35513
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:35513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43575/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790454033974695-467-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:00.037719 5594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:00.038714 5594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:00.047087 5600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:00.047097 5599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:00.047260 5594 server_base.cc:1034] running on GCE node
W20260926 20:21:00.047097 5602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:00.047986 5594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:00.049079 5594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:00.050242 5594 hybrid_clock.cc:648] HybridClock initialized: now 1790454060050223 us; error 36 us; skew 500 ppm
I20260926 20:21:00.052263 5594 webserver.cc:460] Webserver started at http://127.0.116.254:36119/ using document root <none> and password file <none>
I20260926 20:21:00.052879 5594 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:00.052956 5594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:00.053170 5594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:00.055233 5594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "c2349cf6cd334dcdbf7bac93d4b45f78"
format_stamp: "Formatted at 2026-09-26 20:21:00 on dist-test-slave-qsm9"
server_key: "6cc4c861c9d8e3bbc40dee8e29c49437"
server_key_iv: "05535807340b711ac01a4e091b8eac1b"
server_key_version: "encryptionkey@0"
I20260926 20:21:00.055867 5594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "c2349cf6cd334dcdbf7bac93d4b45f78"
format_stamp: "Formatted at 2026-09-26 20:21:00 on dist-test-slave-qsm9"
server_key: "6cc4c861c9d8e3bbc40dee8e29c49437"
server_key_iv: "05535807340b711ac01a4e091b8eac1b"
server_key_version: "encryptionkey@0"
I20260926 20:21:00.059623 5594 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:21:00.062263 5608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:00.063833 5594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:21:00.063997 5594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "c2349cf6cd334dcdbf7bac93d4b45f78"
format_stamp: "Formatted at 2026-09-26 20:21:00 on dist-test-slave-qsm9"
server_key: "6cc4c861c9d8e3bbc40dee8e29c49437"
server_key_iv: "05535807340b711ac01a4e091b8eac1b"
server_key_version: "encryptionkey@0"
I20260926 20:21:00.064127 5594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:00.076740 5594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:00.077493 5594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:00.077710 5594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:00.085101 5594 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:35513
I20260926 20:21:00.085124 5660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:35513 every 8 connection(s)
I20260926 20:21:00.086150 5594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:21:00.088127 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 5594
I20260926 20:21:00.088258 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:21:00.088603 467 external_mini_cluster.cc:1428] Setting key 46eee24be3f2c991ee27c4a403eebe1d
I20260926 20:21:00.090969 5661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:00.097229 5661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78: Bootstrap starting.
I20260926 20:21:00.098559 5664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:00.090262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59382 (local address 127.0.116.254:35513)
0926 20:21:00.090973 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:00.090991 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:00.091058 (+ 67us) server_negotiation.cc:407] Connection header received
0926 20:21:00.091836 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:00.091856 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:00.092280 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:00.092659 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:00.093117 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:00.094110 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:00.095074 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:00.095642 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:00.095814 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:00.096721 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:00.096726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:00.096740 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:21:00.097140 (+ 400us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:00.097195 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:21:00.097215 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:00.097326 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:00.097535 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 20:21:00.097800 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":136,"threads_started":1}
I20260926 20:21:00.100030 5661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:00.100960 5661 log.cc:824] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78: Log is configured to *not* fsync() on all Append() calls
I20260926 20:21:00.101898 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:21:00.103659 5661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78: No bootstrap required, opened a new log
I20260926 20:21:00.107414 5661 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2349cf6cd334dcdbf7bac93d4b45f78" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 35513 } }
I20260926 20:21:00.107759 5661 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:00.107849 5661 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2349cf6cd334dcdbf7bac93d4b45f78, State: Initialized, Role: FOLLOWER
I20260926 20:21:00.108326 5661 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2349cf6cd334dcdbf7bac93d4b45f78" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 35513 } }
I20260926 20:21:00.108490 5661 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:21:00.108588 5661 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:21:00.108700 5661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:00.109733 5661 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2349cf6cd334dcdbf7bac93d4b45f78" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 35513 } }
I20260926 20:21:00.110116 5661 leader_election.cc:302] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2349cf6cd334dcdbf7bac93d4b45f78; no voters:
I20260926 20:21:00.110468 5661 leader_election.cc:288] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:21:00.110608 5666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:21:00.110860 5666 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78 [term 1 LEADER]: Becoming Leader. State: Replica: c2349cf6cd334dcdbf7bac93d4b45f78, State: Running, Role: LEADER
I20260926 20:21:00.111179 5666 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2349cf6cd334dcdbf7bac93d4b45f78" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 35513 } }
I20260926 20:21:00.111989 5661 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:21:00.113199 5667 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2349cf6cd334dcdbf7bac93d4b45f78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2349cf6cd334dcdbf7bac93d4b45f78" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 35513 } } }
I20260926 20:21:00.113353 5667 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:00.113791 5675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:21:00.114362 5668 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2349cf6cd334dcdbf7bac93d4b45f78. Latest consensus state: current_term: 1 leader_uuid: "c2349cf6cd334dcdbf7bac93d4b45f78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2349cf6cd334dcdbf7bac93d4b45f78" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 35513 } } }
I20260926 20:21:00.114560 5668 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:00.117637 5675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:21:00.123984 5675 catalog_manager.cc:1347] Generated new cluster ID: 2dcef1c5780b4d29bcedbbc281031135
I20260926 20:21:00.124086 5675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:21:00.145721 5675 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:21:00.147131 5675 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:21:00.153774 5675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2349cf6cd334dcdbf7bac93d4b45f78: Generated new TSK 0
I20260926 20:21:00.154644 5675 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 20:21:00.183123 5691 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 20:21:00.183262 5691 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 20:21:00.183885 5664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:00.165608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59388 (local address 127.0.116.254:35513)
0926 20:21:00.165757 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:00.165762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:00.166043 (+ 281us) server_negotiation.cc:407] Connection header received
0926 20:21:00.166297 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:00.166303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:00.166381 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:00.166474 (+ 93us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:21:00.166832 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:00.167482 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:00.168939 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:00.169188 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:00.169276 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:00.169658 (+ 382us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:21:00.183458 (+ 13800us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 20:21:00.183771 (+ 313us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59388: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 20:21:00.183974 5664 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59388: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 20:21:00.185406 5690 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:00.165512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:35513 (local address 127.0.0.1:59388)
0926 20:21:00.165921 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:00.165937 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:00.166083 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:00.166525 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:00.166529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:00.166552 (+ 23us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:21:00.166681 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:00.166687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:00.167634 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:00.167638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:00.168725 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:00.168738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:00.169307 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:00.169310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:00.169404 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:00.169459 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:21:00.183777 (+ 14318us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 20:21:00.184052 (+ 275us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.116.254:35513: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":138,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.116.254:35513: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 20:21:00.186337 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 5594
2026-09-26T20:21:00Z chronyd exiting
I20260926 20:21:00.224222 467 test_util.cc:175] -----------------------------------------------
I20260926 20:21:00.224318 467 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790454033974695-467-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (324 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 20:21:00.225381 467 mini_oidc.cc:152] Starting JWKS server
I20260926 20:21:00.226267 467 webserver.cc:460] Webserver started at https://127.0.0.1:39371/ using document root <none> and password file <none>
I20260926 20:21:00.226356 467 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 20:21:00.226595 467 webserver.cc:460] Webserver started at http://127.0.0.1:34497/ using document root <none> and password file <none>
2026-09-26T20:21:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:21:00Z Disabled control of system clock
I20260926 20:21:00.236420 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:32919
--webserver_interface=127.0.116.254
--webserver_port=0
--builtin_ntp_servers=127.0.116.212:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:32919
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39371/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790454033974695-467-0/testchainca.cert with env {}
W20260926 20:21:00.350787 5697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:00.351132 5697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:00.351208 5697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:00.355399 5697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:21:00.355547 5697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:00.355579 5697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:00.355618 5697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:21:00.355656 5697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:21:00.355695 5697 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:21:00.355726 5697 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39371/jwks.json
I20260926 20:21:00.362046 5697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:32919
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:32919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39371/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790454033974695-467-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:00.363245 5697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:00.364410 5697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:00.373473 5702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:00.373473 5705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:00.373620 5697 server_base.cc:1034] running on GCE node
W20260926 20:21:00.373827 5703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:00.374450 5697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:00.375741 5697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:00.376968 5697 hybrid_clock.cc:648] HybridClock initialized: now 1790454060376950 us; error 35 us; skew 500 ppm
I20260926 20:21:00.379226 5697 webserver.cc:460] Webserver started at http://127.0.116.254:45239/ using document root <none> and password file <none>
I20260926 20:21:00.379944 5697 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:00.380015 5697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:00.380236 5697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:00.382478 5697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "bc78b14fae62477da8d89b99e9efaa4a"
format_stamp: "Formatted at 2026-09-26 20:21:00 on dist-test-slave-qsm9"
server_key: "39f39662e4a498557c193f1044e6f8a4"
server_key_iv: "ea42fa19a4ce044225f68622ae1aa939"
server_key_version: "encryptionkey@0"
I20260926 20:21:00.383086 5697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "bc78b14fae62477da8d89b99e9efaa4a"
format_stamp: "Formatted at 2026-09-26 20:21:00 on dist-test-slave-qsm9"
server_key: "39f39662e4a498557c193f1044e6f8a4"
server_key_iv: "ea42fa19a4ce044225f68622ae1aa939"
server_key_version: "encryptionkey@0"
I20260926 20:21:00.387138 5697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:21:00.389950 5711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:00.391417 5697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:21:00.391623 5697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "bc78b14fae62477da8d89b99e9efaa4a"
format_stamp: "Formatted at 2026-09-26 20:21:00 on dist-test-slave-qsm9"
server_key: "39f39662e4a498557c193f1044e6f8a4"
server_key_iv: "ea42fa19a4ce044225f68622ae1aa939"
server_key_version: "encryptionkey@0"
I20260926 20:21:00.391775 5697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:00.417434 5697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:00.418203 5697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:00.418416 5697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:00.425750 5697 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:32919
I20260926 20:21:00.425772 5763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:32919 every 8 connection(s)
I20260926 20:21:00.426769 5697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:21:00.429867 5764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:00.433490 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 5697
I20260926 20:21:00.433624 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:21:00.433945 467 external_mini_cluster.cc:1428] Setting key 13d9bc48ce8eb27f5633153a6eccd28e
I20260926 20:21:00.436053 5764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a: Bootstrap starting.
I20260926 20:21:00.438325 5764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:00.439078 5764 log.cc:824] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:21:00.441386 5764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a: No bootstrap required, opened a new log
I20260926 20:21:00.444389 5767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:00.435617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50104 (local address 127.0.116.254:32919)
0926 20:21:00.436382 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:00.436420 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:00.436458 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:21:00.437119 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:00.437135 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:00.437556 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:00.437907 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:00.438370 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:00.439489 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:00.440899 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:00.441373 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:00.441580 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:00.442517 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:00.442522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:00.442537 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:21:00.442935 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:00.442996 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:21:00.443015 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:00.443095 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:00.443265 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 20:21:00.443630 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":195,"threads_started":1}
I20260926 20:21:00.445537 5764 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc78b14fae62477da8d89b99e9efaa4a" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 32919 } }
I20260926 20:21:00.445778 5764 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:00.445842 5764 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc78b14fae62477da8d89b99e9efaa4a, State: Initialized, Role: FOLLOWER
I20260926 20:21:00.446292 5764 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc78b14fae62477da8d89b99e9efaa4a" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 32919 } }
I20260926 20:21:00.446466 5764 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:21:00.446578 5764 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:21:00.446714 5764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:00.447602 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:21:00.447973 5764 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc78b14fae62477da8d89b99e9efaa4a" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 32919 } }
I20260926 20:21:00.448351 5764 leader_election.cc:302] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc78b14fae62477da8d89b99e9efaa4a; no voters:
I20260926 20:21:00.448750 5764 leader_election.cc:288] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:21:00.448873 5769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:21:00.449093 5769 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a [term 1 LEADER]: Becoming Leader. State: Replica: bc78b14fae62477da8d89b99e9efaa4a, State: Running, Role: LEADER
I20260926 20:21:00.449476 5769 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc78b14fae62477da8d89b99e9efaa4a" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 32919 } }
I20260926 20:21:00.450157 5764 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:21:00.451116 5771 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc78b14fae62477da8d89b99e9efaa4a. Latest consensus state: current_term: 1 leader_uuid: "bc78b14fae62477da8d89b99e9efaa4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc78b14fae62477da8d89b99e9efaa4a" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 32919 } } }
I20260926 20:21:00.451292 5771 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:00.451972 5770 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc78b14fae62477da8d89b99e9efaa4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc78b14fae62477da8d89b99e9efaa4a" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 32919 } } }
I20260926 20:21:00.452073 5770 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:00.452330 5778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:21:00.455765 5778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:21:00.461772 5778 catalog_manager.cc:1347] Generated new cluster ID: 308f4b12213843b1b2ceb6ff15afdeaa
I20260926 20:21:00.461858 5778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:21:00.473980 5778 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:21:00.475152 5778 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:21:00.480118 5778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc78b14fae62477da8d89b99e9efaa4a: Generated new TSK 0
I20260926 20:21:00.480983 5778 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 20:21:00.509016 5794 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 20:21:00.509119 5794 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 20:21:00.509852 5767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:00.492449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50112 (local address 127.0.116.254:32919)
0926 20:21:00.492621 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:00.492627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:00.492850 (+ 223us) server_negotiation.cc:407] Connection header received
0926 20:21:00.493038 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:00.493041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:00.493097 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:00.493203 (+ 106us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:21:00.493484 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:00.494112 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:00.495364 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:00.495718 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:00.495854 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:00.496175 (+ 321us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:21:00.509482 (+ 13307us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 20:21:00.509733 (+ 251us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50112: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 20:21:00.509949 5767 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50112: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 20:21:00.510116 5793 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:00.492372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:32919 (local address 127.0.0.1:50112)
0926 20:21:00.492728 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:00.492744 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:00.492874 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:00.493224 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:00.493228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:00.493237 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:21:00.493365 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:00.493372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:00.494233 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:00.494237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:00.495232 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:00.495243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:00.495864 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:00.495868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:00.495962 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:00.495994 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:21:00.509711 (+ 13717us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 20:21:00.509919 (+ 208us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.116.254:32919: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":156,"threads_started":1}
I20260926 20:21:00.510788 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 5697
2026-09-26T20:21:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (324 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 20:21:00.549952 467 mini_oidc.cc:152] Starting JWKS server
I20260926 20:21:00.550856 467 webserver.cc:460] Webserver started at https://127.0.0.1:40351/ using document root <none> and password file <none>
I20260926 20:21:00.550957 467 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 20:21:00.551179 467 webserver.cc:460] Webserver started at http://127.0.0.1:45241/ using document root <none> and password file <none>
2026-09-26T20:21:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:21:00Z Disabled control of system clock
I20260926 20:21:00.560873 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:34685
--webserver_interface=127.0.116.254
--webserver_port=0
--builtin_ntp_servers=127.0.116.212:39417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:34685
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40351/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 20:21:00.676038 5800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:00.676301 5800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:00.676340 5800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:00.680321 5800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:21:00.680377 5800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:00.680392 5800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:00.680409 5800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:21:00.680424 5800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:21:00.680440 5800 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:21:00.680461 5800 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40351/jwks.json
I20260926 20:21:00.685242 5800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:39417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:34685
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:34685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40351/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:00.686316 5800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:00.687701 5800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:21:00.696877 5800 server_base.cc:1034] running on GCE node
W20260926 20:21:00.696823 5805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:00.696843 5808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:00.696817 5806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:00.697706 5800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:00.698912 5800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:00.700105 5800 hybrid_clock.cc:648] HybridClock initialized: now 1790454060700088 us; error 29 us; skew 500 ppm
I20260926 20:21:00.702143 5800 webserver.cc:460] Webserver started at http://127.0.116.254:45499/ using document root <none> and password file <none>
I20260926 20:21:00.702786 5800 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:00.702883 5800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:00.703137 5800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:00.705374 5800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "33abfa62212243a58836b7895376dbfc"
format_stamp: "Formatted at 2026-09-26 20:21:00 on dist-test-slave-qsm9"
server_key: "b42852a53ae6bdbf465b318281e3fc95"
server_key_iv: "2f3050cfe6bb7ce975c5e7d80578ef2f"
server_key_version: "encryptionkey@0"
I20260926 20:21:00.705992 5800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "33abfa62212243a58836b7895376dbfc"
format_stamp: "Formatted at 2026-09-26 20:21:00 on dist-test-slave-qsm9"
server_key: "b42852a53ae6bdbf465b318281e3fc95"
server_key_iv: "2f3050cfe6bb7ce975c5e7d80578ef2f"
server_key_version: "encryptionkey@0"
I20260926 20:21:00.710196 5800 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:21:00.713135 5814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:00.714588 5800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:21:00.714759 5800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "33abfa62212243a58836b7895376dbfc"
format_stamp: "Formatted at 2026-09-26 20:21:00 on dist-test-slave-qsm9"
server_key: "b42852a53ae6bdbf465b318281e3fc95"
server_key_iv: "2f3050cfe6bb7ce975c5e7d80578ef2f"
server_key_version: "encryptionkey@0"
I20260926 20:21:00.714908 5800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:00.734174 5800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:00.734949 5800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:00.735168 5800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:00.742336 5800 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:34685
I20260926 20:21:00.742348 5866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:34685 every 8 connection(s)
I20260926 20:21:00.743332 5800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:21:00.746620 5867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:00.747350 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 5800
I20260926 20:21:00.747507 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:21:00.747823 467 external_mini_cluster.cc:1428] Setting key 9e02788f10cc97956c711ba8abc9d6bf
I20260926 20:21:00.753019 5867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc: Bootstrap starting.
I20260926 20:21:00.755879 5867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:00.756722 5867 log.cc:824] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc: Log is configured to *not* fsync() on all Append() calls
I20260926 20:21:00.758095 5870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:00.749299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34094 (local address 127.0.116.254:34685)
0926 20:21:00.749994 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:00.750009 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:00.750046 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:21:00.750706 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:00.750723 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:00.751219 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:00.751602 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:00.752104 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:00.753045 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:00.754283 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:00.754838 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:00.754986 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:00.756068 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:00.756079 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:00.756095 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:21:00.756567 (+ 472us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:00.756630 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:21:00.756653 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:00.756739 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:00.756936 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 20:21:00.757270 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":149,"threads_started":1}
I20260926 20:21:00.758826 5867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc: No bootstrap required, opened a new log
I20260926 20:21:00.761471 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:21:00.763322 5867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33abfa62212243a58836b7895376dbfc" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34685 } }
I20260926 20:21:00.763628 5867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:00.763701 5867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33abfa62212243a58836b7895376dbfc, State: Initialized, Role: FOLLOWER
I20260926 20:21:00.764137 5867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33abfa62212243a58836b7895376dbfc" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34685 } }
I20260926 20:21:00.764281 5867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:21:00.764351 5867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:21:00.764451 5867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:00.765419 5867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33abfa62212243a58836b7895376dbfc" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34685 } }
I20260926 20:21:00.765766 5867 leader_election.cc:302] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33abfa62212243a58836b7895376dbfc; no voters:
I20260926 20:21:00.766084 5867 leader_election.cc:288] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:21:00.766240 5872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:21:00.766510 5872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc [term 1 LEADER]: Becoming Leader. State: Replica: 33abfa62212243a58836b7895376dbfc, State: Running, Role: LEADER
I20260926 20:21:00.766903 5872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33abfa62212243a58836b7895376dbfc" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34685 } }
I20260926 20:21:00.767385 5867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:21:00.768520 5873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33abfa62212243a58836b7895376dbfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33abfa62212243a58836b7895376dbfc" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34685 } } }
I20260926 20:21:00.768654 5873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:00.768889 5874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33abfa62212243a58836b7895376dbfc. Latest consensus state: current_term: 1 leader_uuid: "33abfa62212243a58836b7895376dbfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33abfa62212243a58836b7895376dbfc" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34685 } } }
I20260926 20:21:00.769002 5874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:00.769168 5881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:21:00.772449 5881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:21:00.778370 5881 catalog_manager.cc:1347] Generated new cluster ID: d46c05229e474d49a0809612094bb404
I20260926 20:21:00.778473 5881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:21:00.804628 5881 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:21:00.805693 5881 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:21:00.816154 5881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33abfa62212243a58836b7895376dbfc: Generated new TSK 0
I20260926 20:21:00.816987 5881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:21:00.841722 5870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:00.836107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34106 (local address 127.0.116.254:34685)
0926 20:21:00.836261 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:00.836267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:00.836439 (+ 172us) server_negotiation.cc:407] Connection header received
0926 20:21:00.836598 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:00.836601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:00.836662 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:00.836787 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:21:00.837058 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:00.837892 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:00.838861 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:00.839218 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:00.839320 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:00.839687 (+ 367us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:21:00.841299 (+ 1612us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40351/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 20:21:00.841553 (+ 254us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34106: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40351/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 20:21:00.841799 5870 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34106: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40351/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 20:21:00.843850 5896 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:00.836019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:34685 (local address 127.0.0.1:34106)
0926 20:21:00.836345 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:00.836359 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:00.836474 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:00.836790 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:00.836792 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:00.836798 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:21:00.836902 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:00.836950 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:00.838039 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:00.838042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:00.838741 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:00.838749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:00.839343 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:00.839346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:00.839417 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:00.839446 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:21:00.841494 (+ 2048us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40351/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 20:21:00.843658 (+ 2164us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.116.254:34685: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40351/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":1854,"thread_start_us":119,"threads_started":1}
I20260926 20:21:00.844904 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 5800
E20260926 20:21:00.853605 5897 webserver.cc:533] Webserver: sslize failed:
E20260926 20:21:00.853704 5897 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T20:21:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (333 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T20:21:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:21:00Z Disabled control of system clock
I20260926 20:21:00.892266 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:41077
--webserver_interface=127.0.116.254
--webserver_port=0
--builtin_ntp_servers=127.0.116.212:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:41077
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790454033974695-467-0/testchainca.cert with env {}
W20260926 20:21:01.008723 5901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:01.009016 5901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:01.009056 5901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:01.013154 5901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:21:01.013219 5901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:01.013235 5901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:01.013252 5901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:21:01.013267 5901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:21:01.013325 5901 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:21:01.013352 5901 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 20:21:01.019027 5901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:41077
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:41077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790454033974695-467-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:01.020176 5901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:01.021195 5901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:01.029951 5906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:01.029994 5909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:01.030004 5907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:01.030164 5901 server_base.cc:1034] running on GCE node
I20260926 20:21:01.030889 5901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:01.032243 5901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:01.033437 5901 hybrid_clock.cc:648] HybridClock initialized: now 1790454061033402 us; error 46 us; skew 500 ppm
I20260926 20:21:01.035372 5901 webserver.cc:460] Webserver started at http://127.0.116.254:45705/ using document root <none> and password file <none>
I20260926 20:21:01.035992 5901 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:01.036055 5901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:01.036262 5901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:01.038357 5901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "b9a8a4fed2bb4f33936d5176c4cf1f63"
format_stamp: "Formatted at 2026-09-26 20:21:01 on dist-test-slave-qsm9"
server_key: "1baac34786d70d22617df8fd44b6ac03"
server_key_iv: "f9b063dbf92b0622c19a987532f58449"
server_key_version: "encryptionkey@0"
I20260926 20:21:01.038926 5901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "b9a8a4fed2bb4f33936d5176c4cf1f63"
format_stamp: "Formatted at 2026-09-26 20:21:01 on dist-test-slave-qsm9"
server_key: "1baac34786d70d22617df8fd44b6ac03"
server_key_iv: "f9b063dbf92b0622c19a987532f58449"
server_key_version: "encryptionkey@0"
I20260926 20:21:01.042613 5901 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 20:21:01.045282 5915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:01.046626 5901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:21:01.046725 5901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "b9a8a4fed2bb4f33936d5176c4cf1f63"
format_stamp: "Formatted at 2026-09-26 20:21:01 on dist-test-slave-qsm9"
server_key: "1baac34786d70d22617df8fd44b6ac03"
server_key_iv: "f9b063dbf92b0622c19a987532f58449"
server_key_version: "encryptionkey@0"
I20260926 20:21:01.046867 5901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:01.064217 5901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:01.064956 5901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:01.065156 5901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:01.072420 5901 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:41077
I20260926 20:21:01.072453 5967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:41077 every 8 connection(s)
I20260926 20:21:01.073433 5901 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:21:01.076968 5968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:01.078531 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 5901
I20260926 20:21:01.078655 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:21:01.078997 467 external_mini_cluster.cc:1428] Setting key 3180e96dacfd27084b57d2d76e9c8629
I20260926 20:21:01.083786 5968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63: Bootstrap starting.
I20260926 20:21:01.086342 5968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:01.087181 5968 log.cc:824] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63: Log is configured to *not* fsync() on all Append() calls
I20260926 20:21:01.088816 5971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:01.080542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48742 (local address 127.0.116.254:41077)
0926 20:21:01.081239 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:01.081254 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:01.081290 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:21:01.081897 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:01.081914 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:01.082315 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:01.082652 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:01.083120 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:01.084195 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:01.085070 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:01.085542 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:01.085716 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:01.086682 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:01.086687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:01.086700 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:21:01.087087 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:01.087143 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:21:01.087162 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:01.087271 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:01.087431 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 20:21:01.087764 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":138,"threads_started":1}
I20260926 20:21:01.089577 5968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63: No bootstrap required, opened a new log
I20260926 20:21:01.092051 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:21:01.094020 5968 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a8a4fed2bb4f33936d5176c4cf1f63" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41077 } }
I20260926 20:21:01.094260 5968 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:01.094326 5968 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9a8a4fed2bb4f33936d5176c4cf1f63, State: Initialized, Role: FOLLOWER
I20260926 20:21:01.094827 5968 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a8a4fed2bb4f33936d5176c4cf1f63" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41077 } }
I20260926 20:21:01.094998 5968 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:21:01.095072 5968 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:21:01.095208 5968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:01.096312 5968 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a8a4fed2bb4f33936d5176c4cf1f63" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41077 } }
I20260926 20:21:01.096694 5968 leader_election.cc:302] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9a8a4fed2bb4f33936d5176c4cf1f63; no voters:
I20260926 20:21:01.096999 5968 leader_election.cc:288] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:21:01.097184 5973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:21:01.097486 5973 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63 [term 1 LEADER]: Becoming Leader. State: Replica: b9a8a4fed2bb4f33936d5176c4cf1f63, State: Running, Role: LEADER
I20260926 20:21:01.097829 5973 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a8a4fed2bb4f33936d5176c4cf1f63" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41077 } }
I20260926 20:21:01.098438 5968 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:21:01.099690 5975 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9a8a4fed2bb4f33936d5176c4cf1f63. Latest consensus state: current_term: 1 leader_uuid: "b9a8a4fed2bb4f33936d5176c4cf1f63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a8a4fed2bb4f33936d5176c4cf1f63" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41077 } } }
I20260926 20:21:01.099823 5975 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:01.100124 5974 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9a8a4fed2bb4f33936d5176c4cf1f63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a8a4fed2bb4f33936d5176c4cf1f63" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41077 } } }
I20260926 20:21:01.100250 5974 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:01.100378 5982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:21:01.104074 5982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:21:01.110666 5982 catalog_manager.cc:1347] Generated new cluster ID: 3146d25dd00740569d1dd362749d7401
I20260926 20:21:01.110823 5982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:21:01.129789 5982 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:21:01.130839 5982 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:21:01.144474 5982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9a8a4fed2bb4f33936d5176c4cf1f63: Generated new TSK 0
I20260926 20:21:01.145270 5982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:21:01.211690 5971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:01.164443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48750 (local address 127.0.116.254:41077)
0926 20:21:01.164571 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:01.164576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:01.164729 (+ 153us) server_negotiation.cc:407] Connection header received
0926 20:21:01.164930 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:01.164933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:01.164981 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:01.165102 (+ 121us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:21:01.165364 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:01.165917 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:01.166798 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:01.167040 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:01.167159 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:01.167477 (+ 318us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:21:01.211113 (+ 43636us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 20:21:01.211471 (+ 358us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48750: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 20:21:01.211791 5971 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48750: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 20:21:01.211874 5997 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:01.164348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41077 (local address 127.0.0.1:48750)
0926 20:21:01.164638 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:01.164651 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:01.164776 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:01.165108 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:01.165111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:01.165119 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:21:01.165237 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:01.165249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:01.166031 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:01.166038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:01.166677 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:01.166685 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:01.167180 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:01.167184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:01.167272 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:01.167300 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:21:01.211370 (+ 44070us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 20:21:01.211649 (+ 279us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.116.254:41077: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":99,"threads_started":1}
I20260926 20:21:01.212591 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 5901
2026-09-26T20:21:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (347 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (124 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (130 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:21:01 dist-test-slave-qsm9 krb5kdc[6009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:21:01 dist-test-slave-qsm9 krb5kdc[6009](info): set up 2 sockets
Sep 26 20:21:01 dist-test-slave-qsm9 krb5kdc[6009](info): commencing operation
krb5kdc: starting...
W20260926 20:21:02.516412 467 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:21:02 dist-test-slave-qsm9 krb5kdc[6009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:21:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:21:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:21:02.648773 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:44115
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:40889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:44115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:02.765694 6022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:02.766042 6022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:02.766124 6022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:02.770332 6022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:21:02.770448 6022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:02.770488 6022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:02.770529 6022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:21:02.770560 6022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:21:02.776260 6022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:40889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:44115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:44115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:02.777431 6022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:02.778491 6022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:02.787422 6027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:02.787609 6022 server_base.cc:1034] running on GCE node
W20260926 20:21:02.787475 6028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:02.787392 6030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:02.788340 6022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:02.789490 6022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:02.790661 6022 hybrid_clock.cc:648] HybridClock initialized: now 1790454062790629 us; error 46 us; skew 500 ppm
Sep 26 20:21:02 dist-test-slave-qsm9 krb5kdc[6009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:02.799319 6022 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:21:02.800388 6022 webserver.cc:460] Webserver started at http://127.0.116.254:44771/ using document root <none> and password file <none>
I20260926 20:21:02.801009 6022 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:02.801075 6022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:02.801306 6022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:02.803531 6022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "0f7300d86f52406e9526a81d91f4f76d"
format_stamp: "Formatted at 2026-09-26 20:21:02 on dist-test-slave-qsm9"
server_key: "68f6c6882a868f9a5e92ac6c1b0ffea7"
server_key_iv: "e64830f471342206f353e60b333ef5ab"
server_key_version: "encryptionkey@0"
I20260926 20:21:02.804153 6022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "0f7300d86f52406e9526a81d91f4f76d"
format_stamp: "Formatted at 2026-09-26 20:21:02 on dist-test-slave-qsm9"
server_key: "68f6c6882a868f9a5e92ac6c1b0ffea7"
server_key_iv: "e64830f471342206f353e60b333ef5ab"
server_key_version: "encryptionkey@0"
I20260926 20:21:02.808143 6022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:21:02.810726 6037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:02.812143 6022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:21:02.812304 6022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "0f7300d86f52406e9526a81d91f4f76d"
format_stamp: "Formatted at 2026-09-26 20:21:02 on dist-test-slave-qsm9"
server_key: "68f6c6882a868f9a5e92ac6c1b0ffea7"
server_key_iv: "e64830f471342206f353e60b333ef5ab"
server_key_version: "encryptionkey@0"
I20260926 20:21:02.812428 6022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:02.839346 6022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:02.842849 6022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:02.843086 6022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:02.850666 6022 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:44115
I20260926 20:21:02.850980 6089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:44115 every 8 connection(s)
I20260926 20:21:02.851775 6022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:21:02.854672 6090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:02.856310 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 6022
I20260926 20:21:02.856480 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:21:02.856858 467 external_mini_cluster.cc:1428] Setting key 42dceca200aca5b074b886463125d48d
I20260926 20:21:02.861812 6090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d: Bootstrap starting.
I20260926 20:21:02.864661 6090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:02.865625 6090 log.cc:824] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:21:02 dist-test-slave-qsm9 krb5kdc[6009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:21:02.868153 6090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d: No bootstrap required, opened a new log
I20260926 20:21:02.873356 6090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7300d86f52406e9526a81d91f4f76d" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44115 } }
I20260926 20:21:02.873659 6090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:02.873731 6090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f7300d86f52406e9526a81d91f4f76d, State: Initialized, Role: FOLLOWER
I20260926 20:21:02.874064 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:02.858433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.0.116.254:44115)
0926 20:21:02.858888 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:02.858910 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:02.858957 (+ 47us) server_negotiation.cc:407] Connection header received
0926 20:21:02.859913 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:02.859943 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:02.860412 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:02.860836 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:02.861870 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:02.863358 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:02.864357 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:02.865134 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:02.865292 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:02.868381 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:02.868387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:02.868405 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:02.868437 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:02.870519 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:02.871031 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:02.871041 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:02.871048 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:02.871223 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:02.871529 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:02.871532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:02.871534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:02.872157 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:02.872394 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:02.872753 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 20:21:02.873070 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":106,"threads_started":1}
I20260926 20:21:02.874316 6090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7300d86f52406e9526a81d91f4f76d" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44115 } }
I20260926 20:21:02.874485 6090 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:21:02.874545 6090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:21:02.874645 6090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:02.875917 6090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7300d86f52406e9526a81d91f4f76d" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44115 } }
I20260926 20:21:02.876291 6090 leader_election.cc:302] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f7300d86f52406e9526a81d91f4f76d; no voters:
I20260926 20:21:02.876665 6090 leader_election.cc:288] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:21:02.876793 6095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:21:02.877099 6095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d [term 1 LEADER]: Becoming Leader. State: Replica: 0f7300d86f52406e9526a81d91f4f76d, State: Running, Role: LEADER
I20260926 20:21:02.877501 6095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7300d86f52406e9526a81d91f4f76d" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44115 } }
I20260926 20:21:02.877990 6090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:21:02.879243 6097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f7300d86f52406e9526a81d91f4f76d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7300d86f52406e9526a81d91f4f76d" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44115 } } }
I20260926 20:21:02.879385 6097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:02.879748 6096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f7300d86f52406e9526a81d91f4f76d. Latest consensus state: current_term: 1 leader_uuid: "0f7300d86f52406e9526a81d91f4f76d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7300d86f52406e9526a81d91f4f76d" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44115 } } }
I20260926 20:21:02.879911 6096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:02.879828 6104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:21:02.883378 6104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:21:02.889627 6104 catalog_manager.cc:1347] Generated new cluster ID: 9362a2987625484da0805d8f8c3d3da1
I20260926 20:21:02.889722 6104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:21:02.905258 6104 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:21:02.906714 6104 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:21:02.915306 6104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f7300d86f52406e9526a81d91f4f76d: Generated new TSK 0
I20260926 20:21:02.916334 6104 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
I20260926 20:21:02.954984 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44115
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:40889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:03.071477 6116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:03.071827 6116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:03.071872 6116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:03.076339 6116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:03.076414 6116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:03.076479 6116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:21:03.082304 6116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:40889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:03.083416 6116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:03.084535 6116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:03.094908 6124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:03.094908 6122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:03.094918 6121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:03.095331 6116 server_base.cc:1034] running on GCE node
I20260926 20:21:03.095793 6116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:03.096460 6116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:03.097677 6116 hybrid_clock.cc:648] HybridClock initialized: now 1790454063097658 us; error 28 us; skew 500 ppm
Sep 26 20:21:03 dist-test-slave-qsm9 krb5kdc[6009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:03.100627 6116 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:21:03.101590 6116 webserver.cc:460] Webserver started at http://127.0.116.193:43905/ using document root <none> and password file <none>
I20260926 20:21:03.102128 6116 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:03.102187 6116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:03.102351 6116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:03.104457 6116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "0629c3e96de04fd0bf8a796fe1d2c25b"
format_stamp: "Formatted at 2026-09-26 20:21:03 on dist-test-slave-qsm9"
server_key: "d695ead8ddd3f8cc16497593b038356f"
server_key_iv: "86915d5c5cca10bcbde54d23c0d5068c"
server_key_version: "encryptionkey@0"
I20260926 20:21:03.105001 6116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "0629c3e96de04fd0bf8a796fe1d2c25b"
format_stamp: "Formatted at 2026-09-26 20:21:03 on dist-test-slave-qsm9"
server_key: "d695ead8ddd3f8cc16497593b038356f"
server_key_iv: "86915d5c5cca10bcbde54d23c0d5068c"
server_key_version: "encryptionkey@0"
I20260926 20:21:03.108757 6116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:21:03.111232 6131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:03.112743 6116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:21:03.112874 6116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "0629c3e96de04fd0bf8a796fe1d2c25b"
format_stamp: "Formatted at 2026-09-26 20:21:03 on dist-test-slave-qsm9"
server_key: "d695ead8ddd3f8cc16497593b038356f"
server_key_iv: "86915d5c5cca10bcbde54d23c0d5068c"
server_key_version: "encryptionkey@0"
I20260926 20:21:03.113051 6116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:03.130085 6116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:03.133304 6116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:03.133505 6116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:03.134146 6116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:03.135000 6116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:03.135046 6116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:03.135113 6116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:03.135144 6116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:03.144781 6116 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:41035
I20260926 20:21:03.144804 6244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:41035 every 8 connection(s)
I20260926 20:21:03.145721 6116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:21:03.151880 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 6116
I20260926 20:21:03.152022 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:21:03.152282 467 external_mini_cluster.cc:1428] Setting key fcbfc0f2f7f9d2e63c635fb99a121f45
Sep 26 20:21:03 dist-test-slave-qsm9 krb5kdc[6009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:21:03.162524 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:03.147867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:39707 (local address 127.0.116.254:44115)
0926 20:21:03.148016 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:03.148020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:03.148904 (+ 884us) server_negotiation.cc:407] Connection header received
0926 20:21:03.150342 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:03.150346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:03.150405 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:03.150482 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:03.151816 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:03.152502 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:03.154199 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:03.154454 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:03.154575 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:03.157804 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:03.157812 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:03.157818 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:03.157869 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:03.159764 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:03.160587 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:03.160592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:03.160593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:03.160701 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:03.161086 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:03.161091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:03.161096 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:03.161380 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:03.161506 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:03.162062 (+ 556us) server_negotiation.cc:299] Negotiation successful
0926 20:21:03.162256 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.0.116.194@KRBTEST.COM; defaulting to no policy
I20260926 20:21:03.163470 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:03.148260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44115 (local address 127.0.116.193:39707)
0926 20:21:03.148738 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:03.148782 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:03.150119 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:03.150666 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:03.150674 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:03.151055 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:03.151647 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:03.151665 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:03.152640 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:03.152643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:03.154064 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:03.154075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:03.154625 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:03.154629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:03.154831 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:03.155566 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:03.155593 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:03.157595 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:03.159905 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:03.159911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:03.159927 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:03.160443 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:03.160826 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:03.160829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:03.160830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:03.160975 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:03.161514 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:03.161520 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:03.161791 (+ 271us) client_negotiation.cc:769] Sending connection context
0926 20:21:03.161965 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 20:21:03.162323 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":123,"threads_started":1}
I20260926 20:21:03.165048 6245 heartbeater.cc:347] Connected to a master server at 127.0.116.254:44115
I20260926 20:21:03.165398 6245 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.194@KRBTEST.COM" created.
I20260926 20:21:03.166138 6245 heartbeater.cc:511] Master 127.0.116.254:44115 requested a full tablet report, sending...
I20260926 20:21:03.168207 6050 ts_manager.cc:194] Registered new tserver with Master: 0629c3e96de04fd0bf8a796fe1d2c25b (127.0.116.193:41035)
I20260926 20:21:03.169643 6050 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:39707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
I20260926 20:21:03.186721 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44115
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:40889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:03.303952 6250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:03.304258 6250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:03.304335 6250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:03.308632 6250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:03.308712 6250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:03.308813 6250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:21:03.314823 6250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:40889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:03.316326 6250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:03.317559 6250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:03.327706 6255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:03.327706 6258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:03.327872 6256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:03.328018 6250 server_base.cc:1034] running on GCE node
I20260926 20:21:03.328543 6250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:03.329224 6250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:03.330441 6250 hybrid_clock.cc:648] HybridClock initialized: now 1790454063330421 us; error 32 us; skew 500 ppm
Sep 26 20:21:03 dist-test-slave-qsm9 krb5kdc[6009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:03.333570 6250 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:21:03.334445 6250 webserver.cc:460] Webserver started at http://127.0.116.194:33669/ using document root <none> and password file <none>
I20260926 20:21:03.334985 6250 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:03.335043 6250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:03.335214 6250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:03.337312 6250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "962f2074f2f74a85ae58039a871dd846"
format_stamp: "Formatted at 2026-09-26 20:21:03 on dist-test-slave-qsm9"
server_key: "00faa2562b0e90241b3096f88f9f61b2"
server_key_iv: "f713ffd4752dd0f8556354c12a460a53"
server_key_version: "encryptionkey@0"
I20260926 20:21:03.337821 6250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "962f2074f2f74a85ae58039a871dd846"
format_stamp: "Formatted at 2026-09-26 20:21:03 on dist-test-slave-qsm9"
server_key: "00faa2562b0e90241b3096f88f9f61b2"
server_key_iv: "f713ffd4752dd0f8556354c12a460a53"
server_key_version: "encryptionkey@0"
I20260926 20:21:03.341580 6250 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:21:03.344501 6265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:03.345835 6250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 20:21:03.345961 6250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "962f2074f2f74a85ae58039a871dd846"
format_stamp: "Formatted at 2026-09-26 20:21:03 on dist-test-slave-qsm9"
server_key: "00faa2562b0e90241b3096f88f9f61b2"
server_key_iv: "f713ffd4752dd0f8556354c12a460a53"
server_key_version: "encryptionkey@0"
I20260926 20:21:03.346088 6250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:03.360617 6250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:03.364289 6250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:03.364461 6250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:03.365031 6250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:03.365851 6250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:03.365901 6250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:03.365943 6250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:03.365962 6250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:03.375314 6250 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:42849
I20260926 20:21:03.375393 6379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:42849 every 8 connection(s)
I20260926 20:21:03.376374 6250 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:21:03.383795 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 6250
I20260926 20:21:03.383934 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:21:03.384223 467 external_mini_cluster.cc:1428] Setting key 2ad0887c0124ba0e311abcd2a5b54b98
Sep 26 20:21:03 dist-test-slave-qsm9 krb5kdc[6009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:21:03.391727 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:03.378386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:58255 (local address 127.0.116.254:44115)
0926 20:21:03.378631 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:03.378638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:03.379635 (+ 997us) server_negotiation.cc:407] Connection header received
0926 20:21:03.381385 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:03.381388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:03.381440 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:03.381527 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:03.382875 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:03.383439 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:03.384869 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:03.385153 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:03.385233 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:03.388140 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:03.388143 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:03.388145 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:03.388170 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:03.389590 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:03.390116 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:03.390120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:03.390122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:03.390205 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:03.390552 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:03.390556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:03.390558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:03.390746 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:03.390854 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:03.391370 (+ 516us) server_negotiation.cc:299] Negotiation successful
0926 20:21:03.391549 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260926 20:21:03.392590 6382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:03.378896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44115 (local address 127.0.116.194:58255)
0926 20:21:03.379433 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:03.379505 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:03.381147 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:03.381745 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:03.381753 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:03.382184 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:03.382702 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:03.382717 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:03.383623 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:03.383626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:03.384754 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:03.384771 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:03.385304 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:03.385308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:03.385524 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:03.386159 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:03.386175 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:03.387923 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:03.389709 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:03.389715 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:03.389731 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:03.390007 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:03.390301 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:03.390305 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:03.390307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:03.390453 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:03.390851 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:03.390858 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:03.391126 (+ 268us) client_negotiation.cc:769] Sending connection context
0926 20:21:03.391318 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 20:21:03.391690 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":159,"threads_started":1}
WARNING: no policy specified for kudu/127.0.116.195@KRBTEST.COM; defaulting to no policy
I20260926 20:21:03.394095 6380 heartbeater.cc:347] Connected to a master server at 127.0.116.254:44115
I20260926 20:21:03.394399 6380 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:03.395129 6380 heartbeater.cc:511] Master 127.0.116.254:44115 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.195@KRBTEST.COM" created.
I20260926 20:21:03.396695 6050 ts_manager.cc:194] Registered new tserver with Master: 962f2074f2f74a85ae58039a871dd846 (127.0.116.194:42849)
I20260926 20:21:03.397667 6050 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:58255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
I20260926 20:21:03.415196 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44115
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:40889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:03.531963 6385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:03.532335 6385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:03.532409 6385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:03.536657 6385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:03.536756 6385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:03.536849 6385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:21:03.542445 6385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:40889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:03.543649 6385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:03.544687 6385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:03.554713 6391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:03.554713 6390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:03.554733 6393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:03.555063 6385 server_base.cc:1034] running on GCE node
I20260926 20:21:03.555567 6385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:03.556227 6385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:03.557413 6385 hybrid_clock.cc:648] HybridClock initialized: now 1790454063557397 us; error 33 us; skew 500 ppm
Sep 26 20:21:03 dist-test-slave-qsm9 krb5kdc[6009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:03.560285 6385 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:21:03.561209 6385 webserver.cc:460] Webserver started at http://127.0.116.195:38117/ using document root <none> and password file <none>
I20260926 20:21:03.561789 6385 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:03.561879 6385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:03.562119 6385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:03.564235 6385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "fec3528ae99d4c0581843196e2279127"
format_stamp: "Formatted at 2026-09-26 20:21:03 on dist-test-slave-qsm9"
server_key: "c0e8180986aa3df75ab8af50e4b6d03e"
server_key_iv: "a17f995aa80e60aad7600ab8984865ad"
server_key_version: "encryptionkey@0"
I20260926 20:21:03.564811 6385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "fec3528ae99d4c0581843196e2279127"
format_stamp: "Formatted at 2026-09-26 20:21:03 on dist-test-slave-qsm9"
server_key: "c0e8180986aa3df75ab8af50e4b6d03e"
server_key_iv: "a17f995aa80e60aad7600ab8984865ad"
server_key_version: "encryptionkey@0"
I20260926 20:21:03.568650 6385 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:21:03.571089 6400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:03.572355 6385 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:21:03.572504 6385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "fec3528ae99d4c0581843196e2279127"
format_stamp: "Formatted at 2026-09-26 20:21:03 on dist-test-slave-qsm9"
server_key: "c0e8180986aa3df75ab8af50e4b6d03e"
server_key_iv: "a17f995aa80e60aad7600ab8984865ad"
server_key_version: "encryptionkey@0"
I20260926 20:21:03.572620 6385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:03.608043 6385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:03.611634 6385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:03.611846 6385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:03.612476 6385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:03.613363 6385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:03.613433 6385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:03.613514 6385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:03.613559 6385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:03.622483 6385 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:40029
I20260926 20:21:03.622505 6514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:40029 every 8 connection(s)
I20260926 20:21:03.623391 6385 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:21:03.632735 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 6385
I20260926 20:21:03.632885 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:21:03.633244 467 external_mini_cluster.cc:1428] Setting key eac23223ac8017dd7092857ace9cfa14
Sep 26 20:21:03 dist-test-slave-qsm9 krb5kdc[6009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:21:03.637779 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:03.625282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:59881 (local address 127.0.116.254:44115)
0926 20:21:03.625435 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:03.625438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:03.626251 (+ 813us) server_negotiation.cc:407] Connection header received
0926 20:21:03.627726 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:03.627729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:03.627776 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:03.627851 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:03.629046 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:03.629604 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:03.630726 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:03.630949 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:03.631067 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:03.634127 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:03.634131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:03.634134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:03.634168 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:03.635912 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:03.636393 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:03.636395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:03.636396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:03.636469 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:03.636764 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:03.636766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:03.636768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:03.637001 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:03.637080 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:03.637560 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 20:21:03.637660 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:21:03.638715 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:03.625659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44115 (local address 127.0.116.195:59881)
0926 20:21:03.626093 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:03.626135 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:03.627451 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:03.628014 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:03.628022 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:03.628377 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:03.628879 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:03.628892 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:03.629721 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:03.629724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:03.630600 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:03.630609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:03.631144 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:03.631156 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:03.631314 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:03.632116 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:03.632133 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:03.633899 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:03.636020 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:03.636025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:03.636040 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:03.636307 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:03.636547 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:03.636549 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:03.636551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:03.636685 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:03.637090 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:03.637096 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:03.637343 (+ 247us) client_negotiation.cc:769] Sending connection context
0926 20:21:03.637511 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 20:21:03.637777 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":113,"threads_started":1}
I20260926 20:21:03.640010 6515 heartbeater.cc:347] Connected to a master server at 127.0.116.254:44115
I20260926 20:21:03.640220 6515 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:03.640661 6515 heartbeater.cc:511] Master 127.0.116.254:44115 requested a full tablet report, sending...
I20260926 20:21:03.641669 6050 ts_manager.cc:194] Registered new tserver with Master: fec3528ae99d4c0581843196e2279127 (127.0.116.195:40029)
I20260926 20:21:03.642195 6050 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:59881
I20260926 20:21:03.647437 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/test-admin.keytab.
Sep 26 20:21:03 dist-test-slave-qsm9 krb5kdc[6009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:03.664387 467 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 20:21:03.667172 6522 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:03.665764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.0.116.193:41035)
0926 20:21:03.666118 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:03.666124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:03.666149 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:21:03.666198 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:03.666203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:03.666395 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:03.666516 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:03.666986 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53896: BlockingRecv error: recv got EOF from 127.0.0.1:53896 (error 108)
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":159,"threads_started":1}
W20260926 20:21:03.667626 467 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/krb5cc' of type FILE
Sep 26 20:21:03 dist-test-slave-qsm9 krb5kdc[6009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:03.668645 467 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 20:21:03 dist-test-slave-qsm9 krb5kdc[6009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:03.669548 467 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 20:21:03 dist-test-slave-qsm9 krb5kdc[6009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.193@KRBTEST.COM
I20260926 20:21:03.679935 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:03.670539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.0.116.193:41035)
0926 20:21:03.670671 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:03.670675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:03.670831 (+ 156us) server_negotiation.cc:407] Connection header received
0926 20:21:03.670993 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:03.670996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:03.671037 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:03.671144 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:03.671730 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:03.672473 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:03.673322 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:03.673713 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:03.673828 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:03.676308 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:03.676318 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:03.676333 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:03.676375 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:03.678090 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:03.678503 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:03.678508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:03.678510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:03.678622 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:03.678933 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:03.678935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:03.678936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:03.679211 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:03.679428 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:03.679678 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 20:21:03.679804 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 20:21:03.683884 6522 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:03.682893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53910 (local address 127.0.116.193:41035)
0926 20:21:03.683033 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:03.683036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:03.683089 (+ 53us) server_negotiation.cc:407] Connection header received
0926 20:21:03.683179 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:03.683181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:03.683217 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:03.683280 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:03.683752 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53910: BlockingRecv error: recv got EOF from 127.0.0.1:53910 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 20:21:03.684165 467 init.cc:244] Error while opening credential cache '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/krb5cc)
Sep 26 20:21:03 dist-test-slave-qsm9 krb5kdc[6009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:03.685221 467 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 20:21:03 dist-test-slave-qsm9 krb5kdc[6009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:03.686164 467 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 20:21:03 dist-test-slave-qsm9 krb5kdc[6009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.193@KRBTEST.COM
I20260926 20:21:03.694770 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:03.687040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53916 (local address 127.0.116.193:41035)
0926 20:21:03.687190 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:03.687194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:03.687207 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:21:03.687306 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:03.687308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:03.687344 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:03.687409 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:03.687963 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:03.688468 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:03.689225 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:03.689499 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:03.689596 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:03.691736 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:03.691742 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:03.691745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:03.691779 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:03.693095 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:03.693578 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:03.693581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:03.693583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:03.693672 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:03.694040 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:03.694042 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:03.694044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:03.694209 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:03.694335 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:03.694545 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 20:21:03.694642 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 20:21:03.697570 6522 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:03.696558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53930 (local address 127.0.116.193:41035)
0926 20:21:03.696695 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:03.696699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:03.696823 (+ 124us) server_negotiation.cc:407] Connection header received
0926 20:21:03.696962 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:03.696965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:03.697008 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:03.697095 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:03.697474 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53930: BlockingRecv error: recv got EOF from 127.0.0.1:53930 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 20:21:03.697710 467 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790454033974695-467-0/krb5kdc/krb5cc' of type FILE
Sep 26 20:21:03 dist-test-slave-qsm9 krb5kdc[6009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:03.698820 467 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 20:21:03 dist-test-slave-qsm9 krb5kdc[6009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:03.699831 467 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 20:21:03 dist-test-slave-qsm9 krb5kdc[6009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.193@KRBTEST.COM
I20260926 20:21:03.708488 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:03.700737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53942 (local address 127.0.116.193:41035)
0926 20:21:03.700864 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:03.700868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:03.700971 (+ 103us) server_negotiation.cc:407] Connection header received
0926 20:21:03.701100 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:03.701108 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:03.701146 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:03.701231 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:03.701720 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:03.702215 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:03.703000 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:03.703216 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:03.703326 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:03.705601 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:03.705604 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:03.705606 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:03.705628 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:03.706879 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:03.707265 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:03.707268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:03.707269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:03.707338 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:03.707637 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:03.707640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:03.707641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:03.707818 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:03.707971 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:03.708182 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 20:21:03.708287 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:21:03.709779 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 6116
I20260926 20:21:03.715553 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 6250
I20260926 20:21:03.721578 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 6385
I20260926 20:21:03.727214 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 6022
2026-09-26T20:21:03Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2262 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:21:03 dist-test-slave-qsm9 krb5kdc[6535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:21:03 dist-test-slave-qsm9 krb5kdc[6535](info): set up 2 sockets
Sep 26 20:21:03 dist-test-slave-qsm9 krb5kdc[6535](info): commencing operation
krb5kdc: starting...
W20260926 20:21:04.824220 467 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:21:04 dist-test-slave-qsm9 krb5kdc[6535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:21:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:21:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.254.keytab.
Entry for principal oryx/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.254.keytab.
I20260926 20:21:04.950748 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:33869
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.254.keytab
--principal=oryx/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:37507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:33869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:05.063480 6548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:05.063833 6548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:05.063928 6548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:05.067956 6548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:21:05.068053 6548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:05.068092 6548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:05.068131 6548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:21:05.068166 6548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:21:05.073496 6548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:37507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:33869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.254.keytab
--principal=oryx/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:33869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:05.074728 6548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:05.075887 6548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:21:05.084774 6548 server_base.cc:1034] running on GCE node
W20260926 20:21:05.084718 6553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:05.084758 6554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:05.084718 6556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:05.085713 6548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:05.086917 6548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:05.088109 6548 hybrid_clock.cc:648] HybridClock initialized: now 1790454065088096 us; error 34 us; skew 500 ppm
Sep 26 20:21:05 dist-test-slave-qsm9 krb5kdc[6535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454065, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:05.091048 6548 init.cc:377] Logged in from keytab as oryx/127.0.116.254@KRBTEST.COM (short username oryx)
I20260926 20:21:05.092168 6548 webserver.cc:460] Webserver started at http://127.0.116.254:33613/ using document root <none> and password file <none>
I20260926 20:21:05.092753 6548 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:05.092845 6548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:05.093075 6548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:05.095250 6548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "72e44190f4b64472a35049cc412c4234"
format_stamp: "Formatted at 2026-09-26 20:21:05 on dist-test-slave-qsm9"
server_key: "442de201dfb2b8fa601c1012c2b7c5ef"
server_key_iv: "097fbeae70a6b3869215ac58d4d262ea"
server_key_version: "encryptionkey@0"
I20260926 20:21:05.095903 6548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "72e44190f4b64472a35049cc412c4234"
format_stamp: "Formatted at 2026-09-26 20:21:05 on dist-test-slave-qsm9"
server_key: "442de201dfb2b8fa601c1012c2b7c5ef"
server_key_iv: "097fbeae70a6b3869215ac58d4d262ea"
server_key_version: "encryptionkey@0"
I20260926 20:21:05.099653 6548 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:21:05.102119 6564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:05.103426 6548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:21:05.103607 6548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "72e44190f4b64472a35049cc412c4234"
format_stamp: "Formatted at 2026-09-26 20:21:05 on dist-test-slave-qsm9"
server_key: "442de201dfb2b8fa601c1012c2b7c5ef"
server_key_iv: "097fbeae70a6b3869215ac58d4d262ea"
server_key_version: "encryptionkey@0"
I20260926 20:21:05.103729 6548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:05.114320 6548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:05.117699 6548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:05.117915 6548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:05.125277 6548 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:33869
I20260926 20:21:05.125556 6616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:33869 every 8 connection(s)
I20260926 20:21:05.126294 6548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:21:05.127393 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 6548
I20260926 20:21:05.127557 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:21:05.127831 467 external_mini_cluster.cc:1428] Setting key 6e07c82bf59892d04a363a38e89defc5
I20260926 20:21:05.129478 6617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:05.134898 6617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234: Bootstrap starting.
Sep 26 20:21:05 dist-test-slave-qsm9 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM
I20260926 20:21:05.138159 6617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:05.139010 6617 log.cc:824] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234: Log is configured to *not* fsync() on all Append() calls
I20260926 20:21:05.141460 6617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234: No bootstrap required, opened a new log
I20260926 20:21:05.142349 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:05.129220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35212 (local address 127.0.116.254:33869)
0926 20:21:05.129666 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:05.129681 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:05.129718 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:21:05.130357 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:05.130373 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:05.130696 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:05.131041 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:05.131864 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:05.132829 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:05.133667 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:05.134138 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:05.134302 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:05.137315 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:05.137320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:05.137332 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:05.137359 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:05.139175 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:05.139681 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:05.139686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:05.139690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:05.139808 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:05.140194 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:05.140197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:05.140199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:05.140694 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:05.140904 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:05.141093 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 20:21:05.141389 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":125,"threads_started":1}
I20260926 20:21:05.145248 6617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } }
I20260926 20:21:05.145498 6617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:05.145537 6617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e44190f4b64472a35049cc412c4234, State: Initialized, Role: FOLLOWER
I20260926 20:21:05.145905 6617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } }
I20260926 20:21:05.146054 6617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:21:05.146128 6617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:21:05.146252 6617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:05.147328 6617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } }
I20260926 20:21:05.147712 6617 leader_election.cc:302] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72e44190f4b64472a35049cc412c4234; no voters:
I20260926 20:21:05.148051 6617 leader_election.cc:288] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:21:05.148214 6622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:21:05.148470 6622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [term 1 LEADER]: Becoming Leader. State: Replica: 72e44190f4b64472a35049cc412c4234, State: Running, Role: LEADER
I20260926 20:21:05.148801 6622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } }
I20260926 20:21:05.149349 6617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:21:05.150918 6624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72e44190f4b64472a35049cc412c4234. Latest consensus state: current_term: 1 leader_uuid: "72e44190f4b64472a35049cc412c4234" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } }
I20260926 20:21:05.150938 6623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72e44190f4b64472a35049cc412c4234" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } }
I20260926 20:21:05.151077 6623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:05.151077 6624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:05.151597 6635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:21:05.154558 6635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:21:05.160727 6635 catalog_manager.cc:1347] Generated new cluster ID: a236558215094ba299536d94a3af8346
I20260926 20:21:05.160826 6635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:21:05.187386 6635 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:21:05.188445 6635 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:21:05.193635 6635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234: Generated new TSK 0
I20260926 20:21:05.194655 6635 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.193.keytab.
Entry for principal oryx/127.0.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.193.keytab.
I20260926 20:21:05.236362 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:33869
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.193.keytab
--principal=oryx/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:37507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:05.353303 6643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:05.353649 6643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:05.353726 6643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:05.358021 6643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:05.358121 6643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:05.358211 6643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:21:05.363875 6643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:37507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.193.keytab
--principal=oryx/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:33869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:05.365039 6643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:05.366128 6643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:05.376212 6651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:05.376343 6643 server_base.cc:1034] running on GCE node
W20260926 20:21:05.376219 6649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:05.376566 6648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:05.376962 6643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:05.377657 6643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:05.378862 6643 hybrid_clock.cc:648] HybridClock initialized: now 1790454065378843 us; error 30 us; skew 500 ppm
Sep 26 20:21:05 dist-test-slave-qsm9 krb5kdc[6535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454065, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:05.381810 6643 init.cc:377] Logged in from keytab as oryx/127.0.116.193@KRBTEST.COM (short username oryx)
I20260926 20:21:05.382723 6643 webserver.cc:460] Webserver started at http://127.0.116.193:42549/ using document root <none> and password file <none>
I20260926 20:21:05.383358 6643 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:05.383451 6643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:05.383708 6643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:05.385764 6643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb"
format_stamp: "Formatted at 2026-09-26 20:21:05 on dist-test-slave-qsm9"
server_key: "4ab3e3a9a74939630d02fd3f0dd13d27"
server_key_iv: "e698950e75f4eb36bbf012ad05bff64c"
server_key_version: "encryptionkey@0"
I20260926 20:21:05.386327 6643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb"
format_stamp: "Formatted at 2026-09-26 20:21:05 on dist-test-slave-qsm9"
server_key: "4ab3e3a9a74939630d02fd3f0dd13d27"
server_key_iv: "e698950e75f4eb36bbf012ad05bff64c"
server_key_version: "encryptionkey@0"
I20260926 20:21:05.390178 6643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 20:21:05.392850 6658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:05.394783 6643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:21:05.394928 6643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb"
format_stamp: "Formatted at 2026-09-26 20:21:05 on dist-test-slave-qsm9"
server_key: "4ab3e3a9a74939630d02fd3f0dd13d27"
server_key_iv: "e698950e75f4eb36bbf012ad05bff64c"
server_key_version: "encryptionkey@0"
I20260926 20:21:05.395095 6643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:05.412147 6643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:05.415047 6643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:05.415198 6643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:05.415813 6643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:05.416594 6643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:05.416639 6643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:05.416679 6643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:05.416703 6643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:05.425387 6643 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:35253
I20260926 20:21:05.425407 6772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:35253 every 8 connection(s)
I20260926 20:21:05.426276 6643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:21:05.433969 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 6643
I20260926 20:21:05.434121 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:21:05.434486 467 external_mini_cluster.cc:1428] Setting key 6099c9838d6313492728d71527fb170d
Sep 26 20:21:05 dist-test-slave-qsm9 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454065, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM
I20260926 20:21:05.440955 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:05.428062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:42307 (local address 127.0.116.254:33869)
0926 20:21:05.428204 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:05.428208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:05.429073 (+ 865us) server_negotiation.cc:407] Connection header received
0926 20:21:05.430362 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:05.430365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:05.430414 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:05.430517 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:05.431692 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:05.432245 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:05.433384 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:05.433635 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:05.433713 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:05.436912 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:05.436916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:05.436918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:05.436940 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:05.438519 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:05.439213 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:05.439216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:05.439217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:05.439297 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:05.439756 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:05.439758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:05.439759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:05.439938 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:05.440016 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:05.440722 (+ 706us) server_negotiation.cc:299] Negotiation successful
0926 20:21:05.440832 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:21:05.442377 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:05.428469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.116.193:42307)
0926 20:21:05.428911 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:05.428954 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:05.430159 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:05.430672 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:05.430679 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:05.431030 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:05.431543 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:05.431558 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:05.432353 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:05.432356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:05.433277 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:05.433287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:05.433801 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:05.433807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:05.434050 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:05.434791 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:05.434814 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:05.436707 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:05.438645 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:05.438655 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:05.438681 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:05.439099 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:05.439400 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:05.439406 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:05.439412 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:05.439657 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:05.440047 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:05.440057 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:05.440437 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 20:21:05.440677 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 20:21:05.441081 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":122,"threads_started":1}
I20260926 20:21:05.443764 6773 heartbeater.cc:347] Connected to a master server at 127.0.116.254:33869
I20260926 20:21:05.444028 6773 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:05.444543 6773 heartbeater.cc:511] Master 127.0.116.254:33869 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.116.194@KRBTEST.COM; defaulting to no policy
I20260926 20:21:05.446095 6581 ts_manager.cc:194] Registered new tserver with Master: dbabb7ce4283472aa26a1dc3d52ee4fb (127.0.116.193:35253)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.116.194@KRBTEST.COM" created.
I20260926 20:21:05.447420 6581 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:42307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.194.keytab.
Entry for principal oryx/127.0.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.194.keytab.
I20260926 20:21:05.464723 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:33869
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.194.keytab
--principal=oryx/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:37507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:05.579711 6778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:05.580014 6778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:05.580094 6778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:05.584340 6778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:05.584434 6778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:05.584530 6778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:21:05.590013 6778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:37507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.194.keytab
--principal=oryx/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:33869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:05.591207 6778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:05.592391 6778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:05.602717 6783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:05.602741 6784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:05.602741 6786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:05.602782 6778 server_base.cc:1034] running on GCE node
I20260926 20:21:05.603469 6778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:05.604202 6778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:05.605394 6778 hybrid_clock.cc:648] HybridClock initialized: now 1790454065605377 us; error 32 us; skew 500 ppm
Sep 26 20:21:05 dist-test-slave-qsm9 krb5kdc[6535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454065, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:05.608435 6778 init.cc:377] Logged in from keytab as oryx/127.0.116.194@KRBTEST.COM (short username oryx)
I20260926 20:21:05.609405 6778 webserver.cc:460] Webserver started at http://127.0.116.194:42355/ using document root <none> and password file <none>
I20260926 20:21:05.609984 6778 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:05.610109 6778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:05.610343 6778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:05.612525 6778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "8a913df7cca14f21980b78e45afb13ed"
format_stamp: "Formatted at 2026-09-26 20:21:05 on dist-test-slave-qsm9"
server_key: "2fc02336d902e1093eac2639f61db5a9"
server_key_iv: "3455fff0988b67d7360c6823cf3398bf"
server_key_version: "encryptionkey@0"
I20260926 20:21:05.613106 6778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "8a913df7cca14f21980b78e45afb13ed"
format_stamp: "Formatted at 2026-09-26 20:21:05 on dist-test-slave-qsm9"
server_key: "2fc02336d902e1093eac2639f61db5a9"
server_key_iv: "3455fff0988b67d7360c6823cf3398bf"
server_key_version: "encryptionkey@0"
I20260926 20:21:05.617123 6778 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:21:05.619908 6793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:05.621393 6778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:21:05.621555 6778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "8a913df7cca14f21980b78e45afb13ed"
format_stamp: "Formatted at 2026-09-26 20:21:05 on dist-test-slave-qsm9"
server_key: "2fc02336d902e1093eac2639f61db5a9"
server_key_iv: "3455fff0988b67d7360c6823cf3398bf"
server_key_version: "encryptionkey@0"
I20260926 20:21:05.621694 6778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:05.643667 6778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:05.646997 6778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:05.647223 6778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:05.647907 6778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:05.648778 6778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:05.648852 6778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:05.648924 6778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:05.648975 6778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:05.658514 6778 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:41615
I20260926 20:21:05.658551 6907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:41615 every 8 connection(s)
I20260926 20:21:05.659456 6778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:21:05.661440 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 6778
I20260926 20:21:05.661574 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:21:05.661852 467 external_mini_cluster.cc:1428] Setting key 05ea091cf328cb2314860c13dc379f83
WARNING: no policy specified for oryx/127.0.116.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.116.195@KRBTEST.COM" created.
Sep 26 20:21:05 dist-test-slave-qsm9 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454065, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM
I20260926 20:21:05.679157 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:05.661417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:49867 (local address 127.0.116.254:33869)
0926 20:21:05.661603 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:05.661608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:05.662538 (+ 930us) server_negotiation.cc:407] Connection header received
0926 20:21:05.664138 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:05.664144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:05.664210 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:05.664304 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:05.665561 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:05.666085 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:05.667246 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:05.667477 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:05.667643 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:05.675188 (+ 7545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:05.675191 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:05.675193 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:05.675224 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:05.676833 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:05.677465 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:05.677468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:05.677469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:05.677552 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:05.677930 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:05.677934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:05.677936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:05.678159 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:05.678269 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:05.678839 (+ 570us) server_negotiation.cc:299] Negotiation successful
0926 20:21:05.678978 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 20:21:05.680132 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:05.661868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.116.194:49867)
0926 20:21:05.662369 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:05.662423 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:05.663841 (+ 1418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:05.664465 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:05.664472 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:05.664873 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:05.665408 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:05.665421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:05.666206 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:05.666213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:05.667120 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:05.667128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:05.667681 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:05.667685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:05.667855 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:05.672515 (+ 4660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:05.672535 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:05.674935 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:05.676995 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:05.677003 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:05.677025 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:05.677332 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:05.677654 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:05.677658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:05.677660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:05.677829 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:05.678277 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:05.678287 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:05.678581 (+ 294us) client_negotiation.cc:769] Sending connection context
0926 20:21:05.678789 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 20:21:05.679158 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":136,"threads_started":1}
I20260926 20:21:05.681378 6908 heartbeater.cc:347] Connected to a master server at 127.0.116.254:33869
I20260926 20:21:05.681641 6908 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:05.682188 6908 heartbeater.cc:511] Master 127.0.116.254:33869 requested a full tablet report, sending...
I20260926 20:21:05.683296 6581 ts_manager.cc:194] Registered new tserver with Master: 8a913df7cca14f21980b78e45afb13ed (127.0.116.194:41615)
I20260926 20:21:05.684052 6581 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:49867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.195.keytab.
Entry for principal oryx/127.0.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.195.keytab.
I20260926 20:21:05.692090 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:33869
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.195.keytab
--principal=oryx/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:37507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:05.809564 6913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:05.809844 6913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:05.809894 6913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:05.814034 6913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:05.814112 6913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:05.814173 6913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:21:05.819641 6913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:37507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.195.keytab
--principal=oryx/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:33869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:05.820693 6913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:05.821651 6913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:05.831898 6921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:05.831883 6918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:05.832024 6913 server_base.cc:1034] running on GCE node
W20260926 20:21:05.831907 6919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:05.832683 6913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:05.833432 6913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:05.834646 6913 hybrid_clock.cc:648] HybridClock initialized: now 1790454065834626 us; error 35 us; skew 500 ppm
Sep 26 20:21:05 dist-test-slave-qsm9 krb5kdc[6535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454065, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:05.837796 6913 init.cc:377] Logged in from keytab as oryx/127.0.116.195@KRBTEST.COM (short username oryx)
I20260926 20:21:05.838913 6913 webserver.cc:460] Webserver started at http://127.0.116.195:35165/ using document root <none> and password file <none>
I20260926 20:21:05.839555 6913 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:05.839646 6913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:05.839901 6913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:05.842509 6913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "65fda696e9f1430abd4889d32512860f"
format_stamp: "Formatted at 2026-09-26 20:21:05 on dist-test-slave-qsm9"
server_key: "bdfbd0fd632ff27bb02c53be687eb3a6"
server_key_iv: "10dc419e6fa9a79e698af21193428d2a"
server_key_version: "encryptionkey@0"
I20260926 20:21:05.843119 6913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "65fda696e9f1430abd4889d32512860f"
format_stamp: "Formatted at 2026-09-26 20:21:05 on dist-test-slave-qsm9"
server_key: "bdfbd0fd632ff27bb02c53be687eb3a6"
server_key_iv: "10dc419e6fa9a79e698af21193428d2a"
server_key_version: "encryptionkey@0"
I20260926 20:21:05.847201 6913 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:21:05.850106 6928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:05.851649 6913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:21:05.851805 6913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "65fda696e9f1430abd4889d32512860f"
format_stamp: "Formatted at 2026-09-26 20:21:05 on dist-test-slave-qsm9"
server_key: "bdfbd0fd632ff27bb02c53be687eb3a6"
server_key_iv: "10dc419e6fa9a79e698af21193428d2a"
server_key_version: "encryptionkey@0"
I20260926 20:21:05.851936 6913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:05.874128 6913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:05.877583 6913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:05.877807 6913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:05.878473 6913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:05.879421 6913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:05.879520 6913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:05.879581 6913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:05.879611 6913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:05.889154 6913 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:37643
I20260926 20:21:05.889165 7043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:37643 every 8 connection(s)
I20260926 20:21:05.890108 6913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:21:05.899435 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 6913
I20260926 20:21:05.899611 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:21:05.899996 467 external_mini_cluster.cc:1428] Setting key 97d1fad74905d8519a0679944254998c
Sep 26 20:21:05 dist-test-slave-qsm9 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454065, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM
I20260926 20:21:05.910230 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:05.891986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:37745 (local address 127.0.116.254:33869)
0926 20:21:05.892126 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:05.892130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:05.892995 (+ 865us) server_negotiation.cc:407] Connection header received
0926 20:21:05.894344 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:05.894347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:05.894442 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:05.894564 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:05.896221 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:05.897305 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:05.900646 (+ 3341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:05.901013 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:05.901116 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:05.904988 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:05.904994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:05.904997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:05.905034 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:05.907065 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:05.907760 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:05.907764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:05.907766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:05.907868 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:05.908469 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:05.908473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:05.908475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:05.908784 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:05.908907 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:05.909840 (+ 933us) server_negotiation.cc:299] Negotiation successful
0926 20:21:05.909984 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:21:05.911294 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:05.892365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.116.195:37745)
0926 20:21:05.892823 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:05.892868 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:05.894121 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:05.894810 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:05.894824 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:05.895391 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:05.896019 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:05.896051 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:05.897441 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:05.897449 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:05.900500 (+ 3051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:05.900513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:05.901192 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:05.901218 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:05.901553 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:05.902227 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:05.902254 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:05.904756 (+ 2502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:05.907248 (+ 2492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:05.907254 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:05.907269 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:05.907644 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:05.908003 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:05.908009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:05.908012 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:05.908320 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:05.908923 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:05.908933 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:05.909334 (+ 401us) client_negotiation.cc:769] Sending connection context
0926 20:21:05.909593 (+ 259us) client_negotiation.cc:240] Negotiation successful
0926 20:21:05.910035 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":118,"threads_started":1}
I20260926 20:21:05.912761 7044 heartbeater.cc:347] Connected to a master server at 127.0.116.254:33869
I20260926 20:21:05.913151 7044 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:05.913861 7044 heartbeater.cc:511] Master 127.0.116.254:33869 requested a full tablet report, sending...
I20260926 20:21:05.915395 6581 ts_manager.cc:194] Registered new tserver with Master: 65fda696e9f1430abd4889d32512860f (127.0.116.195:37643)
I20260926 20:21:05.916039 6581 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:37745
I20260926 20:21:05.927409 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.116.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.116.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.116.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.253.keytab.
Entry for principal oryx/127.0.116.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.253.keytab.
I20260926 20:21:05.962565 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.253:36501
--webserver_interface=127.0.116.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.253.keytab
--principal=oryx/127.0.116.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:37507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:33869,127.0.116.253:36501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:06.080452 7049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:06.080796 7049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:06.080885 7049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:06.085400 7049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:21:06.085500 7049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:06.085542 7049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:06.085613 7049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:21:06.085671 7049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:21:06.091224 7049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:37507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:33869,127.0.116.253:36501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.253.keytab
--principal=oryx/127.0.116.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.253:36501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.116.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
W20260926 20:21:06.091534 7049 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.116.254:33869,127.0.116.253:36501'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 20:21:06.092505 7049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:06.093648 7049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:06.102342 7054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:06.102384 7055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:06.102358 7057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:06.102514 7049 server_base.cc:1034] running on GCE node
I20260926 20:21:06.103222 7049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:06.104449 7049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:06.105647 7049 hybrid_clock.cc:648] HybridClock initialized: now 1790454066105625 us; error 37 us; skew 500 ppm
Sep 26 20:21:06 dist-test-slave-qsm9 krb5kdc[6535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:06.108803 7049 init.cc:377] Logged in from keytab as oryx/127.0.116.253@KRBTEST.COM (short username oryx)
I20260926 20:21:06.109783 7049 webserver.cc:460] Webserver started at http://127.0.116.253:37097/ using document root <none> and password file <none>
I20260926 20:21:06.110384 7049 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:06.110489 7049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:06.110729 7049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:06.112915 7049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/data/instance:
uuid: "00e0d5ec1e01475a871d550cf2f981e0"
format_stamp: "Formatted at 2026-09-26 20:21:06 on dist-test-slave-qsm9"
server_key: "54047cfbc3b30d175935f56d5bef0fe1"
server_key_iv: "849bea31df7b93cc1271a3fbbf714d28"
server_key_version: "encryptionkey@0"
I20260926 20:21:06.113507 7049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/wal/instance:
uuid: "00e0d5ec1e01475a871d550cf2f981e0"
format_stamp: "Formatted at 2026-09-26 20:21:06 on dist-test-slave-qsm9"
server_key: "54047cfbc3b30d175935f56d5bef0fe1"
server_key_iv: "849bea31df7b93cc1271a3fbbf714d28"
server_key_version: "encryptionkey@0"
I20260926 20:21:06.117395 7049 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 20:21:06.120081 7064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:06.121493 7049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:21:06.121644 7049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/wal
uuid: "00e0d5ec1e01475a871d550cf2f981e0"
format_stamp: "Formatted at 2026-09-26 20:21:06 on dist-test-slave-qsm9"
server_key: "54047cfbc3b30d175935f56d5bef0fe1"
server_key_iv: "849bea31df7b93cc1271a3fbbf714d28"
server_key_version: "encryptionkey@0"
I20260926 20:21:06.121769 7049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:06.134087 7049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:06.137351 7049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:06.137571 7049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:06.144428 7049 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.253:36501
I20260926 20:21:06.144709 7116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.253:36501 every 8 connection(s)
I20260926 20:21:06.145452 7049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/data/info.pb
I20260926 20:21:06.148365 7117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:06.149520 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 7049
I20260926 20:21:06.152591 7117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:21:06 dist-test-slave-qsm9 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.253@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM
Sep 26 20:21:06 dist-test-slave-qsm9 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.116.253@KRBTEST.COM
I20260926 20:21:06.164662 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.154476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35226 (local address 127.0.116.254:33869)
0926 20:21:06.154635 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:06.154641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:06.154903 (+ 262us) server_negotiation.cc:407] Connection header received
0926 20:21:06.155016 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:06.155020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:06.155076 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:06.155177 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:06.155961 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.156745 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.157785 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.158145 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.158263 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.160921 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:06.160925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:06.160927 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:06.160959 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:06.162570 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.163084 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.163087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.163089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.163176 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.163535 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.163542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.163544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.163766 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:06.163876 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:06.164377 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 20:21:06.164491 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:21:06.165603 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.151202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49866 (local address 127.0.116.253:36501)
0926 20:21:06.151893 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:06.151916 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:06.151968 (+ 52us) server_negotiation.cc:407] Connection header received
0926 20:21:06.152728 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:06.152749 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:06.153169 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:06.153550 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:06.154463 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.156096 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.157157 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.157673 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.157868 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.161372 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:06.161381 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:06.161396 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:06.161423 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:06.163196 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.163700 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.163706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.163708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.163787 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.164061 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.164066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.164067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.164290 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:06.164506 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:06.164691 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 20:21:06.164824 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":136,"threads_started":1}
I20260926 20:21:06.165602 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.154372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.0.1:35226)
0926 20:21:06.154764 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:06.154813 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:06.154932 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:06.155228 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:06.155232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:06.155665 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:06.155823 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.155832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.156869 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.156872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.157660 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.157670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.158300 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.158304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.158426 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.158891 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:06.158913 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:06.160752 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:06.162685 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.162696 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.162707 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.162960 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.163268 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.163273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.163275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.163407 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.163883 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:06.163892 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:06.164140 (+ 248us) client_negotiation.cc:769] Sending connection context
0926 20:21:06.164322 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 20:21:06.164641 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":128,"threads_started":1}
I20260926 20:21:06.167637 7117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:21:06 dist-test-slave-qsm9 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.253@KRBTEST.COM for oryx/127.0.116.253@KRBTEST.COM
I20260926 20:21:06.175071 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.168189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49882 (local address 127.0.116.253:36501)
0926 20:21:06.168289 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:06.168292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:06.168303 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:21:06.168381 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:06.168383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:06.168413 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:06.168474 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:06.168797 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.169607 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.170293 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.170501 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.170571 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.172385 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:06.172388 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:06.172390 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:06.172409 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:06.173788 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.174146 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.174148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.174149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.174214 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.174428 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.174430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.174431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.174582 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:06.174652 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:06.174856 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 20:21:06.174937 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:21:06.175071 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.168111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:36501 (local address 127.0.0.1:49882)
0926 20:21:06.168225 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:06.168240 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:06.168314 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:06.168500 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:06.168502 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:06.168629 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:06.168712 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.168717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.169696 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.169699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.170204 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.170211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.170587 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.170590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.170655 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.171056 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:06.171066 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:06.172272 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:06.173880 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.173883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.173884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.174056 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.174287 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.174289 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.174290 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.174359 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.174657 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:06.174659 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:06.174728 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 20:21:06.174804 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 20:21:06.174895 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 20:21:06.178061 7117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0: Bootstrap starting.
I20260926 20:21:06.180532 7117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:06.181288 7117 log.cc:824] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0: Log is configured to *not* fsync() on all Append() calls
I20260926 20:21:06.183285 7117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0: No bootstrap required, opened a new log
I20260926 20:21:06.186269 7117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } }
I20260926 20:21:06.186517 7117 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:06.186615 7117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e0d5ec1e01475a871d550cf2f981e0, State: Initialized, Role: FOLLOWER
I20260926 20:21:06.187067 7117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } }
I20260926 20:21:06.187642 7123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } } }
I20260926 20:21:06.187747 7123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:21:06.188373 7117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 20:21:06.193207 7134 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:21:06.193310 7134 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:21:06.195254 7049 master_runner.cc:428] Detected that this master 00e0d5ec1e01475a871d550cf2f981e0 is joining an existing cluster
I20260926 20:21:06.195335 7049 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.116.253:36501
I20260926 20:21:06.206411 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.198256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49890 (local address 127.0.116.253:36501)
0926 20:21:06.198362 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:06.198368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:06.198457 (+ 89us) server_negotiation.cc:407] Connection header received
0926 20:21:06.198547 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:06.198550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:06.198596 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:06.198674 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:06.199221 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.200007 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.201319 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.201660 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.201758 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.202975 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:06.202981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:06.202990 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:06.203024 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:06.204688 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.205209 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.205212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.205214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.205293 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.205580 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.205583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.205585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.205755 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:06.205853 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:06.206151 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 20:21:06.206274 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":30}
I20260926 20:21:06.207007 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.197524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.0.1:35236)
0926 20:21:06.197815 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:06.197829 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:06.197955 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:06.198360 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:06.198363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:06.198652 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:06.198822 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.198829 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.199823 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.199829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.200583 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.200593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.201058 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.201062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.201207 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.202125 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:06.202142 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:06.202575 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:06.204684 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.204687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.204689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.204909 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.205368 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.205370 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.205371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.205431 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.206509 (+ 1078us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:06.206512 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:06.206609 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 20:21:06.206711 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 20:21:06.206832 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":24,"thread_start_us":96,"threads_started":1}
I20260926 20:21:06.206444 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.197937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:36501 (local address 127.0.0.1:49890)
0926 20:21:06.198362 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:06.198376 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:06.198476 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:06.198720 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:06.198724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:06.198936 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:06.199080 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.199087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.200137 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.200140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.201175 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.201185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.201793 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.201799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.201906 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.202362 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:06.202377 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:06.202812 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:06.204801 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.204804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.204806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.205098 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.205377 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.205380 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.205382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.205476 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.205856 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:06.205859 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:06.205961 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 20:21:06.206079 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 20:21:06.206224 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":31,"thread_start_us":96,"threads_started":1}
I20260926 20:21:06.207288 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.197634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35236 (local address 127.0.116.254:33869)
0926 20:21:06.197791 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:06.197799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:06.197911 (+ 112us) server_negotiation.cc:407] Connection header received
0926 20:21:06.198061 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:06.198064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:06.198122 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:06.198212 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:06.198943 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.199707 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.200709 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.200941 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.201053 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.202724 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:06.202728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:06.202730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:06.202756 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:06.204544 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.205014 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.205018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.205019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.205108 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.205520 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.205523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.205525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.205756 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:06.205853 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:06.206963 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0926 20:21:06.207138 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:21:06.212424 7123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:21:06.212596 7123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } }
I20260926 20:21:06.213485 7123 leader_election.cc:288] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72e44190f4b64472a35049cc412c4234 (127.0.116.254:33869)
I20260926 20:21:06.213928 6591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00e0d5ec1e01475a871d550cf2f981e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e44190f4b64472a35049cc412c4234" is_pre_election: true
I20260926 20:21:06.214600 7065 leader_election.cc:302] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 00e0d5ec1e01475a871d550cf2f981e0; no voters: 72e44190f4b64472a35049cc412c4234
I20260926 20:21:06.214869 7123 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 26 20:21:06 dist-test-slave-qsm9 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454065, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.254@KRBTEST.COM for oryx/127.0.116.253@KRBTEST.COM
I20260926 20:21:06.223551 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.212218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49904 (local address 127.0.116.253:36501)
0926 20:21:06.212342 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:06.212346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:06.213197 (+ 851us) server_negotiation.cc:407] Connection header received
0926 20:21:06.213412 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:06.213420 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:06.213487 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:06.213564 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:06.215015 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.215804 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.216669 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.217004 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.217105 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.219756 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:06.219763 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:06.219770 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:06.219802 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:06.221414 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.222105 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.222109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.222110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.222224 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.222575 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.222579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.222581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.222773 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:06.222884 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:06.223238 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 20:21:06.223362 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:21:06.223563 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.212178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:36501 (local address 127.0.0.1:49904)
0926 20:21:06.213065 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:06.213113 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:06.213270 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:06.213763 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:06.213772 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:06.214179 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:06.214306 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.214333 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.215908 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.215911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.216539 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.216547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.217113 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.217116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.217199 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.217775 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:06.217787 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:06.219607 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:06.221540 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.221574 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.221588 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.221975 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.222384 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.222388 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.222390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.222480 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.222885 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:06.222893 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:06.223025 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 20:21:06.223173 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 20:21:06.223385 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":116,"threads_started":1}
I20260926 20:21:06.225402 6581 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.116.253:36501: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0"
member_type: NON_VOTER
last_known_addr {
host: "127.0.116.253"
port: 36501
}
attrs {
promote: true
}
}
dest_uuid: "72e44190f4b64472a35049cc412c4234"
cas_config_opid_index: -1
I20260926 20:21:06.226284 6581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: NON_VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: true } }
I20260926 20:21:06.228833 7146 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 00e0d5ec1e01475a871d550cf2f981e0 (127.0.116.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: NON_VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: true } } }
I20260926 20:21:06.229225 7091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 0 FOLLOWER]: Advancing to term 1
WARNING: no policy specified for oryx/127.0.116.252@KRBTEST.COM; defaulting to no policy
I20260926 20:21:06.230683 7091 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 1 FOLLOWER]: Refusing update from remote peer 72e44190f4b64472a35049cc412c4234: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 20:21:06.230724 7146 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.116.253:36501
I20260926 20:21:06.231309 7049 master.cc:537] Master@127.0.116.253:36501 shutting down...
I20260926 20:21:06.231323 7146 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: NON_VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:21:06.231567 7145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "72e44190f4b64472a35049cc412c4234" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: NON_VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: true } } }
I20260926 20:21:06.231666 7145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.116.252@KRBTEST.COM" created.
W20260926 20:21:06.234328 6565 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 20:21:06.236109 6565 consensus_peers.cc:487] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 -> Peer 00e0d5ec1e01475a871d550cf2f981e0 (127.0.116.253:36501): Couldn't send request to peer 00e0d5ec1e01475a871d550cf2f981e0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 20:21:06.236740 7049 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:21:06.237097 7049 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:21:06.237216 7049 tablet_replica.cc:330] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0: stopping tablet replica
I20260926 20:21:06.241928 7049 master.cc:552] Master@127.0.116.253:36501 shutdown complete.
I20260926 20:21:06.241977 7049 master_runner.cc:305] Clearing existing system tablet
I20260926 20:21:06.242235 7049 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:06.242304 7049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:06.243402 7049 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:21:06.244319 7152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:06.244578 7049 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:06.244724 7049 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:21:06.244769 7049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/wal
uuid: "00e0d5ec1e01475a871d550cf2f981e0"
format_stamp: "Formatted at 2026-09-26 20:21:06 on dist-test-slave-qsm9"
server_key: "54047cfbc3b30d175935f56d5bef0fe1"
server_key_iv: "849bea31df7b93cc1271a3fbbf714d28"
server_key_version: "encryptionkey@0"
I20260926 20:21:06.247002 7049 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:21:06.250507 7049 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 20:21:06.250659 7049 log.cc:1196] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 20:21:06.250969 7049 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0: Deleting consensus metadata
I20260926 20:21:06.251200 7049 master_runner.cc:315] Copying system tablet from 127.0.116.254:33869
I20260926 20:21:06.252525 7049 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0: tablet copy: Beginning tablet copy session from remote peer at address 127.0.116.254:33869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.116.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.252.keytab.
Entry for principal oryx/127.0.116.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.252.keytab.
I20260926 20:21:06.255976 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.252:42181
--webserver_interface=127.0.116.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.252.keytab
--principal=oryx/127.0.116.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:37507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:33869,127.0.116.253:36501,127.0.116.252:42181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:21:06.262856 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.253337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.0.116.254:33869)
0926 20:21:06.253546 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:06.253551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:06.253590 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:21:06.253733 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:06.253736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:06.253789 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:06.253897 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:06.254479 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.255336 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.256782 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.257174 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.257252 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.259358 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:06.259361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:06.259364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:06.259402 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:06.261008 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.261558 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.261561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.261565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.261655 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.261954 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.261958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.261960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.262158 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:06.262266 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:06.262558 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 20:21:06.262672 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 20:21:06.262868 7159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.253193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.0.1:35244)
0926 20:21:06.253498 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:06.253513 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:06.253637 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:06.253967 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:06.253970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:06.254240 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:06.254340 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.254347 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.255486 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.255558 (+ 72us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.256626 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.256639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.257473 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.257477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.257565 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.258677 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:06.258693 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:06.259184 (+ 491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:06.261134 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.261139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.261142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.261450 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.261761 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.261765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.261767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.261856 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.262254 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:06.262257 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:06.262380 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 20:21:06.262496 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 20:21:06.262666 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":94,"threads_started":1}
I20260926 20:21:06.263885 6601 tablet_copy_service.cc:140] P 72e44190f4b64472a35049cc412c4234: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 00e0d5ec1e01475a871d550cf2f981e0 ({username='oryx', principal='oryx/127.0.116.253@KRBTEST.COM'} at 127.0.0.1:35244)
I20260926 20:21:06.264050 6601 tablet_copy_service.cc:161] P 72e44190f4b64472a35049cc412c4234: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 00e0d5ec1e01475a871d550cf2f981e0 at {username='oryx', principal='oryx/127.0.116.253@KRBTEST.COM'} at 127.0.0.1:35244: session id = 00e0d5ec1e01475a871d550cf2f981e0-00000000000000000000000000000000
I20260926 20:21:06.265616 6601 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 20:21:06.267333 7049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 20:21:06.269389 7049 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0: tablet copy: Starting download of 0 data blocks...
I20260926 20:21:06.269629 7049 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0: tablet copy: Starting download of 1 WAL segments...
I20260926 20:21:06.271878 7049 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 20:21:06.273816 6601 tablet_copy_service.cc:342] P 72e44190f4b64472a35049cc412c4234: Request end of tablet copy session 00e0d5ec1e01475a871d550cf2f981e0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.116.253@KRBTEST.COM'} at 127.0.0.1:35244
I20260926 20:21:06.273944 6601 tablet_copy_service.cc:433] P 72e44190f4b64472a35049cc412c4234: ending tablet copy session 00e0d5ec1e01475a871d550cf2f981e0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 00e0d5ec1e01475a871d550cf2f981e0
I20260926 20:21:06.281136 7049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:06.282968 7161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:06.283274 7164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:06.283274 7049 server_base.cc:1034] running on GCE node
W20260926 20:21:06.287710 7162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:06.288026 7049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:06.288401 7049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:06.289583 7049 hybrid_clock.cc:648] HybridClock initialized: now 1790454066289590 us; error 67 us; skew 500 ppm
Sep 26 20:21:06 dist-test-slave-qsm9 krb5kdc[6535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:06.291254 7049 init.cc:377] Logged in from keytab as oryx/127.0.116.253@KRBTEST.COM (short username oryx)
I20260926 20:21:06.291854 7049 webserver.cc:460] Webserver started at http://127.0.116.253:39335/ using document root <none> and password file <none>
I20260926 20:21:06.292085 7049 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:06.292142 7049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:06.293318 7049 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:21:06.294365 7171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:06.294610 7049 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:21:06.294682 7049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/wal
uuid: "00e0d5ec1e01475a871d550cf2f981e0"
format_stamp: "Formatted at 2026-09-26 20:21:06 on dist-test-slave-qsm9"
server_key: "54047cfbc3b30d175935f56d5bef0fe1"
server_key_iv: "849bea31df7b93cc1271a3fbbf714d28"
server_key_version: "encryptionkey@0"
I20260926 20:21:06.294775 7049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:06.319454 7049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:06.321969 7049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:06.327469 7049 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.253:36501
I20260926 20:21:06.327922 7223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.253:36501 every 8 connection(s)
I20260926 20:21:06.328238 7049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-1/data/info.pb
I20260926 20:21:06.329705 7224 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:21:06.330518 7224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0: Bootstrap starting.
I20260926 20:21:06.338699 7224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:21:06.339407 7224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0: Bootstrap complete.
I20260926 20:21:06.339962 7224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: NON_VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: true } }
I20260926 20:21:06.340112 7224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 00e0d5ec1e01475a871d550cf2f981e0, State: Initialized, Role: LEARNER
I20260926 20:21:06.340240 7224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: NON_VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: true } }
I20260926 20:21:06.340601 7227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: NON_VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: true } } }
I20260926 20:21:06.340752 7227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:21:06.340755 7224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:21:06.344808 7238 catalog_manager.cc:1259] Loaded cluster ID: a236558215094ba299536d94a3af8346
I20260926 20:21:06.344887 7238 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0: loading cluster ID for follower catalog manager: success
I20260926 20:21:06.346380 7238 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0: acquiring CA information for follower catalog manager: success
I20260926 20:21:06.347596 7238 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 20:21:06.422883 7160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:06.423185 7160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:06.423230 7160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:06.428411 7160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:21:06.428510 7160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:06.428539 7160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:06.428563 7160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:21:06.428601 7160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:21:06.434546 7160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:37507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:33869,127.0.116.253:36501,127.0.116.252:42181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.252.keytab
--principal=oryx/127.0.116.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.252:42181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.116.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:06.435681 7160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:06.436714 7160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:06.445847 7244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:06.445868 7246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:06.445863 7243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:06.446250 7160 server_base.cc:1034] running on GCE node
I20260926 20:21:06.446857 7160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:06.448179 7160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:06.449362 7160 hybrid_clock.cc:648] HybridClock initialized: now 1790454066449350 us; error 33 us; skew 500 ppm
I20260926 20:21:06.450343 6773 heartbeater.cc:503] Master 127.0.116.254:33869 was elected leader, sending a full tablet report...
Sep 26 20:21:06 dist-test-slave-qsm9 krb5kdc[6535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:06.452847 7160 init.cc:377] Logged in from keytab as oryx/127.0.116.252@KRBTEST.COM (short username oryx)
I20260926 20:21:06.453789 7160 webserver.cc:460] Webserver started at http://127.0.116.252:40771/ using document root <none> and password file <none>
I20260926 20:21:06.454334 7160 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:06.454393 7160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:06.454625 7160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:06.456867 7160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/data/instance:
uuid: "8a4b410506c6418e859f15f84740f6d7"
format_stamp: "Formatted at 2026-09-26 20:21:06 on dist-test-slave-qsm9"
server_key: "c405f66cd80d5ca961c527471f5a2b81"
server_key_iv: "9b8ef54231a36c125d4d726345e8e920"
server_key_version: "encryptionkey@0"
I20260926 20:21:06.457453 7160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/wal/instance:
uuid: "8a4b410506c6418e859f15f84740f6d7"
format_stamp: "Formatted at 2026-09-26 20:21:06 on dist-test-slave-qsm9"
server_key: "c405f66cd80d5ca961c527471f5a2b81"
server_key_iv: "9b8ef54231a36c125d4d726345e8e920"
server_key_version: "encryptionkey@0"
I20260926 20:21:06.461393 7160 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:21:06.464023 7253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:06.465304 7160 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:21:06.465437 7160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/wal
uuid: "8a4b410506c6418e859f15f84740f6d7"
format_stamp: "Formatted at 2026-09-26 20:21:06 on dist-test-slave-qsm9"
server_key: "c405f66cd80d5ca961c527471f5a2b81"
server_key_iv: "9b8ef54231a36c125d4d726345e8e920"
server_key_version: "encryptionkey@0"
I20260926 20:21:06.465552 7160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:06.503604 7160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:06.507323 7160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:06.507556 7160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:06.515751 7160 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.252:42181
I20260926 20:21:06.516043 7306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.252:42181 every 8 connection(s)
I20260926 20:21:06.516906 7160 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/data/info.pb
I20260926 20:21:06.519827 7307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:06.523600 7307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:21:06.525946 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 7160
Sep 26 20:21:06 dist-test-slave-qsm9 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.252@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM
Sep 26 20:21:06 dist-test-slave-qsm9 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.116.252@KRBTEST.COM
I20260926 20:21:06.539736 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.525621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35248 (local address 127.0.116.254:33869)
0926 20:21:06.525810 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:06.525815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:06.526686 (+ 871us) server_negotiation.cc:407] Connection header received
0926 20:21:06.527792 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:06.527795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:06.527847 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:06.527964 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:06.529108 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.529950 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.531427 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.531798 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.531915 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.535122 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:06.535125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:06.535128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:06.535182 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:06.537190 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.537823 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.537828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.537830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.537930 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.538450 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.538454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.538455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.538689 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:06.538797 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:06.539391 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 20:21:06.539553 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 20:21:06.540611 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.525927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.0.1:35248)
0926 20:21:06.526497 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:06.526550 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:06.527589 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:06.528012 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:06.528018 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:06.528576 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:06.528935 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.528962 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.530096 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.530099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.531281 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.531291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.531951 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.531954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.532194 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.532953 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:06.532976 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:06.534967 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:06.537318 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.537326 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.537334 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.537715 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.538140 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.538145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.538149 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.538345 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.538819 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:06.538825 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:06.539104 (+ 279us) client_negotiation.cc:769] Sending connection context
0926 20:21:06.539318 (+ 214us) client_negotiation.cc:240] Negotiation successful
0926 20:21:06.539705 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":62,"spinlock_wait_cycles":768,"thread_start_us":173,"threads_started":1}
I20260926 20:21:06.540617 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.526911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43786 (local address 127.0.116.252:42181)
0926 20:21:06.527264 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:06.527276 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:06.527309 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:21:06.527599 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:06.527620 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:06.527863 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:06.528115 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:06.528990 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.530260 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.531410 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.532102 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.532192 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.535777 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:06.535785 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:06.535799 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:06.535828 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:06.537728 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.538367 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.538372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.538373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.538479 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.538776 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.538779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.538781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.539068 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:06.539212 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:06.539536 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 20:21:06.539940 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":244,"server-negotiator.queue_time_us":259,"spinlock_wait_cycles":2688,"thread_start_us":148,"threads_started":1}
I20260926 20:21:06.542582 7307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:21:06 dist-test-slave-qsm9 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.252@KRBTEST.COM for oryx/127.0.116.253@KRBTEST.COM
I20260926 20:21:06.550676 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.543120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:36501 (local address 127.0.0.1:49910)
0926 20:21:06.543245 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:06.543259 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:06.543338 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:06.543746 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:06.543749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:06.543877 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:06.543972 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.543976 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.544854 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.544856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.545394 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.545401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.545802 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.545804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.545875 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.546351 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:06.546362 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:06.547695 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:06.549229 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.549232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.549233 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.549429 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.549738 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.549741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.549742 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.549824 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.550229 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:06.550231 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:06.550312 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 20:21:06.550403 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 20:21:06.550509 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 20:21:06.550750 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.543234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49910 (local address 127.0.116.253:36501)
0926 20:21:06.543530 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:06.543536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:06.543550 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:21:06.543596 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:06.543598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:06.543637 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:06.543710 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:06.544098 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.544760 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.545493 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.545708 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.545788 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.547831 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:06.547834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:06.547836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:06.547860 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:06.549130 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.549522 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.549528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.549530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.549633 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.549953 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.549956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.549958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.550146 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:06.550221 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:06.550467 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 20:21:06.550608 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":120,"threads_started":1}
I20260926 20:21:06.551462 7307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:21:06 dist-test-slave-qsm9 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.252@KRBTEST.COM for oryx/127.0.116.252@KRBTEST.COM
I20260926 20:21:06.559237 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.552006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43800 (local address 127.0.116.252:42181)
0926 20:21:06.552134 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:06.552138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:06.552172 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:21:06.552309 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:06.552313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:06.552348 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:06.552436 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:06.552782 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.553304 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.554029 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.554285 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.554412 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.556442 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:06.556445 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:06.556448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:06.556493 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:06.557718 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.558120 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.558124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.558126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.558195 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.558451 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.558454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.558456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.558626 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:06.558723 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:06.558953 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 20:21:06.559099 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":72}
I20260926 20:21:06.559237 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.551967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42181 (local address 127.0.0.1:43800)
0926 20:21:06.552115 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:06.552130 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:06.552213 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:06.552443 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:06.552445 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:06.552616 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:06.552676 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.552680 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.553410 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.553416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.553920 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.553927 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.554408 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.554412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.554527 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.554948 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:06.554958 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:06.556310 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:06.557825 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.557830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.557832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.558015 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.558281 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.558285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.558287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.558363 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.558721 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:06.558723 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:06.558801 (+ 78us) client_negotiation.cc:769] Sending connection context
0926 20:21:06.558903 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 20:21:06.559041 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":19}
I20260926 20:21:06.562232 7307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7: Bootstrap starting.
I20260926 20:21:06.564903 7307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:06.565609 7307 log.cc:824] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7: Log is configured to *not* fsync() on all Append() calls
I20260926 20:21:06.567699 7307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7: No bootstrap required, opened a new log
I20260926 20:21:06.570818 7307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } }
I20260926 20:21:06.571036 7307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:06.571070 7307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4b410506c6418e859f15f84740f6d7, State: Initialized, Role: FOLLOWER
I20260926 20:21:06.571532 7307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } }
I20260926 20:21:06.571972 7313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } } }
I20260926 20:21:06.572144 7313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:21:06.572706 7307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 20:21:06.578315 7324 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:21:06.578425 7324 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:21:06.582325 7160 master_runner.cc:428] Detected that this master 8a4b410506c6418e859f15f84740f6d7 is joining an existing cluster
I20260926 20:21:06.582403 7160 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.116.252:42181
I20260926 20:21:06.594286 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.585003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43814 (local address 127.0.116.252:42181)
0926 20:21:06.585109 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:06.585114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:06.585493 (+ 379us) server_negotiation.cc:407] Connection header received
0926 20:21:06.585669 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:06.585672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:06.585713 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:06.585767 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:06.586549 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.587327 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.588901 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.589217 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.589283 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.590784 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:06.590788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:06.590791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:06.590817 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:06.592420 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.592842 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.592845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.592847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.592929 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.593386 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.593389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.593391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.593640 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:06.593735 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:06.594013 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 20:21:06.594122 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":28}
I20260926 20:21:06.594283 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 6643
I20260926 20:21:06.594748 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.585123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42181 (local address 127.0.0.1:43814)
0926 20:21:06.585411 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:06.585425 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:06.585566 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:06.585901 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:06.585904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:06.586231 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:06.586446 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.586454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.587433 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.587436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.588772 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.588785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.589397 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.589400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.589615 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.590281 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:06.590292 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:06.590654 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:06.592535 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.592538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.592539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.592750 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.593151 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.593154 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.593156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.593247 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.593738 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:06.593742 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:06.593847 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 20:21:06.594432 (+ 585us) client_negotiation.cc:240] Negotiation successful
0926 20:21:06.594566 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":127,"spinlock_wait_cycles":5504,"thread_start_us":77,"threads_started":1}
I20260926 20:21:06.601670 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:35253
--local_ip_for_outbound_sockets=127.0.116.193
--tserver_master_addrs=127.0.116.254:33869
--webserver_port=42549
--webserver_interface=127.0.116.193
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.193.keytab
--principal=oryx/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:37507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.116.254:33869,127.0.116.253:36501
--tserver_master_addrs=127.0.116.254:33869,127.0.116.253:36501,127.0.116.252:42181 with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:21:06.620772 7332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.585671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:36501 (local address 127.0.0.1:49920)
0926 20:21:06.585981 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:06.585995 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:06.586120 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:06.586501 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:06.586504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:06.586738 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:06.586855 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.586861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.587937 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.587941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.588832 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.588842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.589498 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.589501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.589663 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.616316 (+ 26653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:06.616339 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:06.616866 (+ 527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:06.618986 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.618990 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.618993 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.619273 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.619627 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.619632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.619636 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.619724 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.620187 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:06.620190 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:06.620285 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 20:21:06.620381 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 20:21:06.620539 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":145,"thread_start_us":80,"threads_started":1}
I20260926 20:21:06.621085 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.585016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.0.116.254:33869)
0926 20:21:06.585137 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:06.585141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:06.585479 (+ 338us) server_negotiation.cc:407] Connection header received
0926 20:21:06.585686 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:06.585690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:06.585742 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:06.585794 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:06.586586 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.587403 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.588702 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.589040 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.589113 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.617017 (+ 27904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:06.617022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:06.617025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:06.617057 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:06.619260 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.619796 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.619800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.619802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.619893 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.620185 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.620189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.620191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.620402 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:06.620489 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:06.620807 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 20:21:06.620928 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:21:06.621335 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.585457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49920 (local address 127.0.116.253:36501)
0926 20:21:06.585570 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:06.585574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:06.586061 (+ 487us) server_negotiation.cc:407] Connection header received
0926 20:21:06.586231 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:06.586234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:06.586287 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:06.586336 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:06.586982 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.587788 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.588947 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.589285 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.589359 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.617078 (+ 27719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:06.617095 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:06.617098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:06.617130 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:06.618834 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.619398 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.619402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.619404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.619526 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.619819 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.619822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.619824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.620049 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:06.620145 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:06.621031 (+ 886us) server_negotiation.cc:299] Negotiation successful
0926 20:21:06.621174 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:21:06.621876 7331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.584953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.0.1:35250)
0926 20:21:06.585382 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:06.585397 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:06.585571 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:06.585934 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:06.585937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:06.586230 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:06.586435 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.586452 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.587578 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.587582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.588563 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.588573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.589292 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.589295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.589593 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.616162 (+ 26569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:06.616185 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:06.616771 (+ 586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:06.619396 (+ 2625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.619400 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.619402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.619673 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.620004 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.620007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.620009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.620090 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.620544 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:06.620546 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:06.620632 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 20:21:06.621277 (+ 645us) client_negotiation.cc:240] Negotiation successful
0926 20:21:06.621694 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":210,"spinlock_wait_cycles":5120,"thread_start_us":114,"threads_started":1}
Sep 26 20:21:06 dist-test-slave-qsm9 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454065, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.254@KRBTEST.COM for oryx/127.0.116.252@KRBTEST.COM
I20260926 20:21:06.635313 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.626029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42181 (local address 127.0.0.1:43828)
0926 20:21:06.626194 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:06.626207 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:06.626303 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:06.626747 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:06.626750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:06.626956 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:06.627070 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.627076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.628113 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.628116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.628930 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.628941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.629507 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.629511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.629606 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.630147 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:06.630162 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:06.631793 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:06.633715 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.633719 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.633721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.633975 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.634273 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.634276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.634278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.634370 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.634773 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:06.634776 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:06.634877 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 20:21:06.634985 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 20:21:06.635142 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260926 20:21:06.635321 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.626115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43828 (local address 127.0.116.252:42181)
0926 20:21:06.626464 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:06.626468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:06.626481 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:21:06.626546 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:06.626549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:06.626619 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:06.626696 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:06.627202 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.627993 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.629053 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.629387 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.629500 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.631941 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:06.631945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:06.631948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:06.631974 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:06.633597 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.634077 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.634087 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.634089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.634176 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.634462 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.634465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.634467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.634673 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:06.634769 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:06.635048 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 20:21:06.635169 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262}
I20260926 20:21:06.636860 6581 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.116.252:42181: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8a4b410506c6418e859f15f84740f6d7"
member_type: NON_VOTER
last_known_addr {
host: "127.0.116.252"
port: 42181
}
attrs {
promote: true
}
}
dest_uuid: "72e44190f4b64472a35049cc412c4234"
cas_config_opid_index: 5
I20260926 20:21:06.637493 6581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: NON_VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: true } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: NON_VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: true } }
I20260926 20:21:06.638598 7146 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8a4b410506c6418e859f15f84740f6d7 (127.0.116.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: NON_VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: true } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: NON_VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: true } } }
I20260926 20:21:06.640717 7146 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.116.252:42181
I20260926 20:21:06.640889 7146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "72e44190f4b64472a35049cc412c4234" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: NON_VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: true } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: NON_VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: true } } }
I20260926 20:21:06.641009 7146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:06.641973 7280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:06.643384 7280 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [term 1 FOLLOWER]: Refusing update from remote peer 72e44190f4b64472a35049cc412c4234: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 20:21:06.643983 7160 master.cc:537] Master@127.0.116.252:42181 shutting down...
I20260926 20:21:06.644528 7145 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: NON_VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 20:21:06.646544 6568 consensus_peers.cc:487] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 -> Peer 8a4b410506c6418e859f15f84740f6d7 (127.0.116.252:42181): Couldn't send request to peer 8a4b410506c6418e859f15f84740f6d7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 20:21:06.650182 7160 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:21:06.650614 7160 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:21:06.650813 7160 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7: stopping tablet replica
I20260926 20:21:06.651530 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.639801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:36501 (local address 127.0.0.1:49936)
0926 20:21:06.646210 (+ 6409us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:06.646230 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:06.646338 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:06.646928 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:06.646931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:06.646950 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:06.647045 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.647051 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.648512 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.648602 (+ 90us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.649868 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.649880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.651047 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.651051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.651147 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.651199 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 20:21:06.651275 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 20:21:06.651356 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6153}
I20260926 20:21:06.651952 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.640092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49936 (local address 127.0.116.253:36501)
0926 20:21:06.640218 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:06.640223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:06.646451 (+ 6228us) server_negotiation.cc:407] Connection header received
0926 20:21:06.646521 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:06.646526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:06.646620 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:06.646709 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:06.647485 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.648344 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.650011 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.650727 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.650826 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.650985 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:06.651757 (+ 772us) server_negotiation.cc:299] Negotiation successful
0926 20:21:06.651837 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:21:06.652398 7198 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 1 LEARNER]: Refusing update from remote peer 72e44190f4b64472a35049cc412c4234: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 20:21:06.652863 7145 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: NON_VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 20:21:06.654743 7198 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8a4b410506c6418e859f15f84740f6d7 (127.0.116.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: NON_VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: true } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: NON_VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: true } } }
I20260926 20:21:06.657243 7227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72e44190f4b64472a35049cc412c4234. Latest consensus state: current_term: 1 leader_uuid: "72e44190f4b64472a35049cc412c4234" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: NON_VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: true } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: NON_VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: true } } }
I20260926 20:21:06.657371 7227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:21:06.657454 7145 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234: attempting to promote NON_VOTER 00e0d5ec1e01475a871d550cf2f981e0 to VOTER
I20260926 20:21:06.657472 7335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "72e44190f4b64472a35049cc412c4234" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: NON_VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: true } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: NON_VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: true } } }
I20260926 20:21:06.657740 7335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:21:06.657943 7145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: NON_VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: true } }
I20260926 20:21:06.659224 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.658917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42181 (local address 127.0.0.1:43834)
0926 20:21:06.659074 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:06.659146 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:42181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 20:21:06.660050 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.659825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42181 (local address 127.0.0.1:43844)
0926 20:21:06.659947 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:06.659990 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:42181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 20:21:06.660153 6568 consensus_peers.cc:487] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 -> Peer 8a4b410506c6418e859f15f84740f6d7 (127.0.116.252:42181): Couldn't send request to peer 8a4b410506c6418e859f15f84740f6d7. Status: Network error: Client connection negotiation failed: client connection to 127.0.116.252:42181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 20:21:06.663723 7198 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 1 LEARNER]: Refusing update from remote peer 72e44190f4b64472a35049cc412c4234: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 20:21:06.664240 7147 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 20:21:06.666059 7146 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 00e0d5ec1e01475a871d550cf2f981e0 (127.0.116.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: NON_VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: true } } }
I20260926 20:21:06.666640 7198 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 00e0d5ec1e01475a871d550cf2f981e0 (127.0.116.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: NON_VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: true } } }
I20260926 20:21:06.667629 7146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "72e44190f4b64472a35049cc412c4234" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: NON_VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: true } } }
I20260926 20:21:06.667745 7146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:06.667856 7146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72e44190f4b64472a35049cc412c4234" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: NON_VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: true } } }
I20260926 20:21:06.667918 7146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:06.668805 7335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "72e44190f4b64472a35049cc412c4234" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: NON_VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: true } } }
I20260926 20:21:06.668967 7335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:21:06.669301 7160 master.cc:552] Master@127.0.116.252:42181 shutdown complete.
I20260926 20:21:06.669368 7160 master_runner.cc:305] Clearing existing system tablet
I20260926 20:21:06.669637 7160 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:06.669691 7160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:06.671003 7160 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 20:21:06.671940 7337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:06.672241 7160 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:06.672359 7160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:21:06.672399 7160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/wal
uuid: "8a4b410506c6418e859f15f84740f6d7"
format_stamp: "Formatted at 2026-09-26 20:21:06 on dist-test-slave-qsm9"
server_key: "c405f66cd80d5ca961c527471f5a2b81"
server_key_iv: "9b8ef54231a36c125d4d726345e8e920"
server_key_version: "encryptionkey@0"
I20260926 20:21:06.674456 7160 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:21:06.676832 7160 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 20:21:06.676923 7160 log.cc:1196] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 20:21:06.677179 7160 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7: Deleting consensus metadata
I20260926 20:21:06.677353 7160 master_runner.cc:315] Copying system tablet from 127.0.116.254:33869
I20260926 20:21:06.678503 7160 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.116.254:33869
I20260926 20:21:06.686173 6908 heartbeater.cc:503] Master 127.0.116.254:33869 was elected leader, sending a full tablet report...
I20260926 20:21:06.689232 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.679137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35262 (local address 127.0.116.254:33869)
0926 20:21:06.679261 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:06.679266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:06.679452 (+ 186us) server_negotiation.cc:407] Connection header received
0926 20:21:06.679581 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:06.679584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:06.679638 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:06.679734 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:06.680248 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.681049 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.682266 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.682621 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.682742 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.684685 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:06.684690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:06.684692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:06.684721 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:06.687181 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.687740 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.687746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.687749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.687845 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.688183 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.688187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.688188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.688513 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:06.688602 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:06.688935 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 20:21:06.689059 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:21:06.690004 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.679038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.0.1:35262)
0926 20:21:06.679339 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:06.679354 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:06.679474 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:06.679769 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:06.679772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:06.679990 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:06.680121 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.680128 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.681183 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.681187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.682149 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.682160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.682746 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.682750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.682838 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.684011 (+ 1173us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:06.684026 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:06.684531 (+ 505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:06.687321 (+ 2790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.687324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.687326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.687621 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.687959 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.687962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.687964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.688060 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.688659 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:06.688661 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:06.688761 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 20:21:06.689666 (+ 905us) client_negotiation.cc:240] Negotiation successful
0926 20:21:06.689827 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":96,"threads_started":1}
I20260926 20:21:06.690704 6601 tablet_copy_service.cc:140] P 72e44190f4b64472a35049cc412c4234: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8a4b410506c6418e859f15f84740f6d7 ({username='oryx', principal='oryx/127.0.116.252@KRBTEST.COM'} at 127.0.0.1:35262)
I20260926 20:21:06.690814 6601 tablet_copy_service.cc:161] P 72e44190f4b64472a35049cc412c4234: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8a4b410506c6418e859f15f84740f6d7 at {username='oryx', principal='oryx/127.0.116.252@KRBTEST.COM'} at 127.0.0.1:35262: session id = 8a4b410506c6418e859f15f84740f6d7-00000000000000000000000000000000
I20260926 20:21:06.691944 6601 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 20:21:06.693301 7160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 20:21:06.695829 7160 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7: tablet copy: Starting download of 0 data blocks...
I20260926 20:21:06.696027 7160 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7: tablet copy: Starting download of 1 WAL segments...
I20260926 20:21:06.697691 7160 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 20:21:06.699827 6601 tablet_copy_service.cc:342] P 72e44190f4b64472a35049cc412c4234: Request end of tablet copy session 8a4b410506c6418e859f15f84740f6d7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.116.252@KRBTEST.COM'} at 127.0.0.1:35262
I20260926 20:21:06.699937 6601 tablet_copy_service.cc:433] P 72e44190f4b64472a35049cc412c4234: ending tablet copy session 8a4b410506c6418e859f15f84740f6d7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8a4b410506c6418e859f15f84740f6d7
I20260926 20:21:06.708673 7160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:06.710906 7344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:06.710968 7345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:06.711200 7347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:06.712038 7160 server_base.cc:1034] running on GCE node
I20260926 20:21:06.712234 7160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:06.712564 7160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:06.713729 7160 hybrid_clock.cc:648] HybridClock initialized: now 1790454066713716 us; error 31 us; skew 500 ppm
Sep 26 20:21:06 dist-test-slave-qsm9 krb5kdc[6535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:06.715479 7160 init.cc:377] Logged in from keytab as oryx/127.0.116.252@KRBTEST.COM (short username oryx)
I20260926 20:21:06.716048 7160 webserver.cc:460] Webserver started at http://127.0.116.252:34449/ using document root <none> and password file <none>
I20260926 20:21:06.716248 7160 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:06.716293 7160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:06.717834 7160 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:21:06.719020 7354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:06.719344 7160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:21:06.719426 7160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/wal
uuid: "8a4b410506c6418e859f15f84740f6d7"
format_stamp: "Formatted at 2026-09-26 20:21:06 on dist-test-slave-qsm9"
server_key: "c405f66cd80d5ca961c527471f5a2b81"
server_key_iv: "9b8ef54231a36c125d4d726345e8e920"
server_key_version: "encryptionkey@0"
I20260926 20:21:06.719591 7160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:06.729279 7160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:06.731680 7160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:06.736982 7160 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.252:42181
I20260926 20:21:06.737015 7406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.252:42181 every 8 connection(s)
I20260926 20:21:06.737655 7160 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/master-2/data/info.pb
I20260926 20:21:06.739475 7407 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:21:06.740774 7407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7: Bootstrap starting.
I20260926 20:21:06.753132 7407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:21:06.753456 7407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7: Bootstrap complete.
I20260926 20:21:06.754000 7407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: NON_VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: true } }
I20260926 20:21:06.754150 7407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8a4b410506c6418e859f15f84740f6d7, State: Initialized, Role: LEARNER
I20260926 20:21:06.754257 7407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: NON_VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: true } }
I20260926 20:21:06.754591 7413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: NON_VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: true } } }
I20260926 20:21:06.754726 7407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 20:21:06.754485 7333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20260926 20:21:06.754794 7413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [sys.catalog]: This master's current role is: LEARNER
W20260926 20:21:06.754841 7333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:06.754908 7333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 20:21:06.757589 7424 catalog_manager.cc:1259] Loaded cluster ID: a236558215094ba299536d94a3af8346
I20260926 20:21:06.757659 7424 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7: loading cluster ID for follower catalog manager: success
I20260926 20:21:06.759842 7424 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7: acquiring CA information for follower catalog manager: success
W20260926 20:21:06.761042 7333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:06.761116 7333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:06.761205 7333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:21:06.761262 7424 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:21:06.766862 7333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:37507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.193.keytab
--principal=oryx/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:35253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=42549
--tserver_master_addrs=127.0.116.254:33869,127.0.116.253:36501,127.0.116.252:42181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:06.768060 7333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:06.769126 7333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:06.779366 7426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:06.779534 7429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:06.779371 7427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:06.779963 7333 server_base.cc:1034] running on GCE node
I20260926 20:21:06.780427 7333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:06.781091 7333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:06.782275 7333 hybrid_clock.cc:648] HybridClock initialized: now 1790454066782241 us; error 45 us; skew 500 ppm
Sep 26 20:21:06 dist-test-slave-qsm9 krb5kdc[6535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:06.785312 7333 init.cc:377] Logged in from keytab as oryx/127.0.116.193@KRBTEST.COM (short username oryx)
I20260926 20:21:06.786221 7333 webserver.cc:460] Webserver started at http://127.0.116.193:42549/ using document root <none> and password file <none>
I20260926 20:21:06.786799 7333 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:06.786881 7333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:06.791117 7333 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:21:06.793427 7436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:06.794833 7333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:21:06.794970 7333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb"
format_stamp: "Formatted at 2026-09-26 20:21:05 on dist-test-slave-qsm9"
server_key: "4ab3e3a9a74939630d02fd3f0dd13d27"
server_key_iv: "e698950e75f4eb36bbf012ad05bff64c"
server_key_version: "encryptionkey@0"
I20260926 20:21:06.795692 7333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:06.813071 7333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:06.816277 7333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:06.816444 7333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:06.817051 7333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:06.817950 7333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:06.817999 7333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:06.818041 7333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:06.818058 7333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:06.828441 7333 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:35253
I20260926 20:21:06.828464 7550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:35253 every 8 connection(s)
I20260926 20:21:06.829610 7333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:21:06.831028 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 7333
I20260926 20:21:06.831254 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 6778
I20260926 20:21:06.838966 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:41615
--local_ip_for_outbound_sockets=127.0.116.194
--tserver_master_addrs=127.0.116.254:33869
--webserver_port=42355
--webserver_interface=127.0.116.194
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.194.keytab
--principal=oryx/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:37507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.116.254:33869,127.0.116.253:36501
--tserver_master_addrs=127.0.116.254:33869,127.0.116.253:36501,127.0.116.252:42181 with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:21:06 dist-test-slave-qsm9 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.252@KRBTEST.COM
Sep 26 20:21:06 dist-test-slave-qsm9 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.253@KRBTEST.COM
Sep 26 20:21:06 dist-test-slave-qsm9 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM
I20260926 20:21:06.849380 7556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.833614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:45787 (local address 127.0.116.252:42181)
0926 20:21:06.834050 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:06.834057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:06.834079 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:21:06.834718 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:06.834723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:06.834777 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:06.834842 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:06.837167 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.838238 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.840177 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.840544 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.840636 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.844488 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:06.844492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:06.844495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:06.844531 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:06.846475 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.847215 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.847219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.847222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.847323 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.847822 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.847825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.847828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.848120 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:06.848213 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:06.848992 (+ 779us) server_negotiation.cc:299] Negotiation successful
0926 20:21:06.849164 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":78,"threads_started":1}
I20260926 20:21:06.849910 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.834268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:35787 (local address 127.0.116.253:36501)
0926 20:21:06.834397 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:06.834404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:06.834655 (+ 251us) server_negotiation.cc:407] Connection header received
0926 20:21:06.834847 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:06.834854 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:06.834914 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:06.834983 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:06.837158 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.838024 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.840550 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.840945 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.841062 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.845956 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:06.845963 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:06.845967 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:06.846002 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:06.847801 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.848265 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.848268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.848270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.848354 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.848680 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.848686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.848690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.848959 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:06.849067 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:06.849528 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 20:21:06.849693 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:21:06.850340 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.832468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:51101 (local address 127.0.116.254:33869)
0926 20:21:06.832650 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:06.832656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:06.834554 (+ 1898us) server_negotiation.cc:407] Connection header received
0926 20:21:06.834747 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:06.834750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:06.834802 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:06.834857 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:06.837172 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.838005 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.840198 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.840538 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.840636 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.845858 (+ 5222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:06.845866 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:06.845871 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:06.845913 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:06.848402 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.848994 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.848999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.849001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.849096 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:06.849400 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:06.849404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:06.849406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:06.849654 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:06.849739 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:06.850079 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 20:21:06.850199 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 20:21:06.851922 7555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.832120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42181 (local address 127.0.116.193:45787)
0926 20:21:06.833285 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:06.833348 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:06.834449 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:06.835295 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:06.835305 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:06.836031 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:06.836961 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.836983 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.838371 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.838374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.840025 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.840036 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.840862 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.840865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.841139 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.842099 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:06.842127 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:06.844247 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:06.846627 (+ 2380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.846637 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.846662 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.847073 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.847442 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.847448 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.847452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.847700 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.848280 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:06.848287 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:06.848680 (+ 393us) client_negotiation.cc:769] Sending connection context
0926 20:21:06.849828 (+ 1148us) client_negotiation.cc:240] Negotiation successful
0926 20:21:06.850636 (+ 808us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":374,"spinlock_wait_cycles":11776,"thread_start_us":728,"threads_started":1}
I20260926 20:21:06.851922 7557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.834140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:36501 (local address 127.0.116.193:35787)
0926 20:21:06.834555 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:06.834571 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:06.834737 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:06.835296 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:06.835306 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:06.836038 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:06.836972 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.836986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.838195 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.838198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.840298 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.840312 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.841323 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.841328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.841435 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.842563 (+ 1128us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:06.842583 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:06.845766 (+ 3183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:06.847919 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.847923 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.847924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.848148 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.848466 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.848468 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.848470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.848548 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.849121 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:06.849182 (+ 61us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:06.849301 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 20:21:06.850271 (+ 970us) client_negotiation.cc:240] Negotiation successful
0926 20:21:06.850755 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":1299,"spinlock_wait_cycles":14592,"thread_start_us":79,"threads_started":1}
I20260926 20:21:06.852216 7558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:06.832120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.116.193:51101)
0926 20:21:06.834458 (+ 2338us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:06.834471 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:06.834640 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:06.835295 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:06.835305 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:06.836040 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:06.836961 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.836983 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.838193 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.838196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.840025 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:06.840036 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:06.840862 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:06.840865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:06.841139 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:06.842994 (+ 1855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:06.843010 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:06.845351 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:06.848654 (+ 3303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.848658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.848660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.848878 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.849215 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:06.849218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:06.849221 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:06.849300 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:06.849795 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:06.849798 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:06.849883 (+ 85us) client_negotiation.cc:769] Sending connection context
0926 20:21:06.850516 (+ 633us) client_negotiation.cc:240] Negotiation successful
0926 20:21:06.850839 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2251,"mutex_wait_us":2181,"spinlock_wait_cycles":11264,"thread_start_us":107,"threads_started":1}
I20260926 20:21:06.853749 7552 heartbeater.cc:347] Connected to a master server at 127.0.116.254:33869
I20260926 20:21:06.853837 7553 heartbeater.cc:347] Connected to a master server at 127.0.116.253:36501
I20260926 20:21:06.854039 7552 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:06.854038 7553 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:06.854657 7552 heartbeater.cc:511] Master 127.0.116.254:33869 requested a full tablet report, sending...
I20260926 20:21:06.854656 7553 heartbeater.cc:511] Master 127.0.116.253:36501 requested a full tablet report, sending...
I20260926 20:21:06.853780 7551 heartbeater.cc:347] Connected to a master server at 127.0.116.252:42181
I20260926 20:21:06.855290 7551 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:06.855484 7551 heartbeater.cc:511] Master 127.0.116.252:42181 requested a full tablet report, sending...
I20260926 20:21:06.856134 6581 ts_manager.cc:194] Re-registered known tserver with Master: dbabb7ce4283472aa26a1dc3d52ee4fb (127.0.116.193:35253)
I20260926 20:21:06.856675 7371 ts_manager.cc:194] Registered new tserver with Master: dbabb7ce4283472aa26a1dc3d52ee4fb (127.0.116.193:35253)
I20260926 20:21:06.856976 6581 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:51101
I20260926 20:21:06.857926 7188 ts_manager.cc:194] Registered new tserver with Master: dbabb7ce4283472aa26a1dc3d52ee4fb (127.0.116.193:35253)
I20260926 20:21:06.918941 7044 heartbeater.cc:503] Master 127.0.116.254:33869 was elected leader, sending a full tablet report...
W20260926 20:21:06.962095 7559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:06.962388 7559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:06.962432 7559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:06.966668 7559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:06.966737 7559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:06.966840 7559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:21:06.972290 7559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:37507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.194.keytab
--principal=oryx/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:41615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=42355
--tserver_master_addrs=127.0.116.254:33869,127.0.116.253:36501,127.0.116.252:42181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:06.973419 7559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:06.974489 7559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:06.985643 7565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:06.985697 7564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:06.985643 7567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:06.985962 7559 server_base.cc:1034] running on GCE node
I20260926 20:21:06.986474 7559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:06.987139 7559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:06.988324 7559 hybrid_clock.cc:648] HybridClock initialized: now 1790454066988289 us; error 52 us; skew 500 ppm
Sep 26 20:21:06 dist-test-slave-qsm9 krb5kdc[6535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:06.991142 7559 init.cc:377] Logged in from keytab as oryx/127.0.116.194@KRBTEST.COM (short username oryx)
I20260926 20:21:06.992129 7559 webserver.cc:460] Webserver started at http://127.0.116.194:42355/ using document root <none> and password file <none>
I20260926 20:21:06.992717 7559 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:06.992825 7559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:06.997053 7559 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:21:06.999238 7574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:07.000737 7559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:21:07.000874 7559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "8a913df7cca14f21980b78e45afb13ed"
format_stamp: "Formatted at 2026-09-26 20:21:05 on dist-test-slave-qsm9"
server_key: "2fc02336d902e1093eac2639f61db5a9"
server_key_iv: "3455fff0988b67d7360c6823cf3398bf"
server_key_version: "encryptionkey@0"
I20260926 20:21:07.001472 7559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:07.038635 7559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:07.042253 7559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:07.042433 7559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:07.043046 7559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:07.043949 7559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:07.043995 7559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:07.044035 7559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:07.044057 7559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:07.053483 7559 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:41615
I20260926 20:21:07.053504 7688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:41615 every 8 connection(s)
I20260926 20:21:07.054637 7559 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:21:07.057231 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 7559
I20260926 20:21:07.057420 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 6913
I20260926 20:21:07.066138 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:37643
--local_ip_for_outbound_sockets=127.0.116.195
--tserver_master_addrs=127.0.116.254:33869
--webserver_port=35165
--webserver_interface=127.0.116.195
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.195.keytab
--principal=oryx/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:37507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.116.254:33869,127.0.116.253:36501
--tserver_master_addrs=127.0.116.254:33869,127.0.116.253:36501,127.0.116.252:42181 with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:21:07 dist-test-slave-qsm9 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM
Sep 26 20:21:07 dist-test-slave-qsm9 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.253@KRBTEST.COM
Sep 26 20:21:07 dist-test-slave-qsm9 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454066, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.252@KRBTEST.COM
I20260926 20:21:07.074615 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.058446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:49147 (local address 127.0.116.253:36501)
0926 20:21:07.058589 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:07.058598 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:07.060002 (+ 1404us) server_negotiation.cc:407] Connection header received
0926 20:21:07.060171 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:07.060174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:07.060231 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:07.060339 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:07.062267 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.063077 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.065107 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.065442 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.065529 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.069641 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:07.069646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:07.069649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:07.069680 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:07.072001 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.072706 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.072711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.072713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.072802 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.073247 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.073251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.073253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.073504 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:07.073584 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:07.074313 (+ 729us) server_negotiation.cc:299] Negotiation successful
0926 20:21:07.074439 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:21:07.074832 7556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.056540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:36219 (local address 127.0.116.252:42181)
0926 20:21:07.056675 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:07.056680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:07.059777 (+ 3097us) server_negotiation.cc:407] Connection header received
0926 20:21:07.060266 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:07.060273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:07.060343 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:07.060400 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:07.062267 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.063078 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.065103 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.065457 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.065528 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.070071 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:07.070078 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:07.070084 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:07.070121 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:07.072245 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.072882 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.072888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.072892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.073001 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.073342 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.073348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.073352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.073607 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:07.073701 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:07.074440 (+ 739us) server_negotiation.cc:299] Negotiation successful
0926 20:21:07.074603 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:21:07.075737 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.057041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.116.194:59921)
0926 20:21:07.058555 (+ 1514us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:07.058617 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:07.059938 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:07.061042 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:07.061047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:07.061293 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:07.062107 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.062120 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.063378 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.063385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.064984 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.064997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.065749 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.065752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.066044 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.067039 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:07.067060 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:07.069782 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:07.072298 (+ 2516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:07.072301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:07.072303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:07.072621 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:07.072912 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:07.072915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:07.072917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:07.073134 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:07.073660 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:07.073670 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:07.074019 (+ 349us) client_negotiation.cc:769] Sending connection context
0926 20:21:07.074237 (+ 218us) client_negotiation.cc:240] Negotiation successful
0926 20:21:07.074583 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1098,"mutex_wait_us":415,"spinlock_wait_cycles":48000,"thread_start_us":717,"threads_started":1}
I20260926 20:21:07.075737 7695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.058343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:36501 (local address 127.0.116.194:49147)
0926 20:21:07.059921 (+ 1578us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:07.059930 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:07.060033 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:07.060523 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:07.060530 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:07.061202 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:07.062097 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.062112 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.063263 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.063267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.064959 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.064970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.065742 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.065746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.066052 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.066898 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:07.066923 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:07.069425 (+ 2502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:07.072157 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:07.072163 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:07.072181 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:07.072584 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:07.072912 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:07.072915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:07.072917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:07.073134 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:07.073890 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:07.073893 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:07.074019 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 20:21:07.074825 (+ 806us) client_negotiation.cc:240] Negotiation successful
0926 20:21:07.074955 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1531,"mutex_wait_us":339,"spinlock_wait_cycles":20352,"thread_start_us":76,"threads_started":1}
I20260926 20:21:07.076472 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.056569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:59921 (local address 127.0.116.254:33869)
0926 20:21:07.056734 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:07.056739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:07.058732 (+ 1993us) server_negotiation.cc:407] Connection header received
0926 20:21:07.060742 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:07.060749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:07.060816 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:07.060878 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:07.062268 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.063231 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.065123 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.065474 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.065564 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.069956 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:07.069962 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:07.069967 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:07.070002 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:07.072186 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.072729 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.072732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.072734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.072812 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.073248 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.073251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.073253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.073497 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:07.073597 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:07.076189 (+ 2592us) server_negotiation.cc:299] Negotiation successful
0926 20:21:07.076329 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 20:21:07.076983 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.057041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42181 (local address 127.0.116.194:36219)
0926 20:21:07.059665 (+ 2624us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:07.059677 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:07.060145 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:07.060577 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:07.060583 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:07.061202 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:07.062091 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.062113 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.063263 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.063267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.064964 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.064970 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.065742 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.065746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.066049 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.067174 (+ 1125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:07.067193 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:07.069869 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:07.072388 (+ 2519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:07.072394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:07.072398 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:07.072755 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:07.073124 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:07.073129 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:07.073131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:07.073225 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:07.073766 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:07.073772 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:07.074218 (+ 446us) client_negotiation.cc:769] Sending connection context
0926 20:21:07.075519 (+ 1301us) client_negotiation.cc:240] Negotiation successful
0926 20:21:07.075692 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2548,"mutex_wait_us":701,"spinlock_wait_cycles":67712,"thread_start_us":169,"threads_started":1}
I20260926 20:21:07.077255 7691 heartbeater.cc:347] Connected to a master server at 127.0.116.253:36501
I20260926 20:21:07.077471 7691 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:07.077596 7689 heartbeater.cc:347] Connected to a master server at 127.0.116.252:42181
I20260926 20:21:07.077656 7689 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:07.077788 7690 heartbeater.cc:347] Connected to a master server at 127.0.116.254:33869
I20260926 20:21:07.077937 7690 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:07.078181 7690 heartbeater.cc:511] Master 127.0.116.254:33869 requested a full tablet report, sending...
I20260926 20:21:07.078152 7691 heartbeater.cc:511] Master 127.0.116.253:36501 requested a full tablet report, sending...
I20260926 20:21:07.078152 7689 heartbeater.cc:511] Master 127.0.116.252:42181 requested a full tablet report, sending...
I20260926 20:21:07.079540 6581 ts_manager.cc:194] Re-registered known tserver with Master: 8a913df7cca14f21980b78e45afb13ed (127.0.116.194:41615)
I20260926 20:21:07.079609 7371 ts_manager.cc:194] Registered new tserver with Master: 8a913df7cca14f21980b78e45afb13ed (127.0.116.194:41615)
I20260926 20:21:07.079618 7188 ts_manager.cc:194] Registered new tserver with Master: 8a913df7cca14f21980b78e45afb13ed (127.0.116.194:41615)
I20260926 20:21:07.080204 6581 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:59921
W20260926 20:21:07.184808 7696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:07.185084 7696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:07.185178 7696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:07.189229 7696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:07.189318 7696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:07.189407 7696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:21:07.194711 7696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:37507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/krb5kdc/oryx_127.0.116.195.keytab
--principal=oryx/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:37643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=35165
--tserver_master_addrs=127.0.116.254:33869,127.0.116.253:36501,127.0.116.252:42181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:07.195946 7696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:07.197011 7696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:07.207271 7702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:07.207310 7701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:07.207370 7696 server_base.cc:1034] running on GCE node
W20260926 20:21:07.207285 7704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:07.208154 7696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:07.208858 7696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:07.210047 7696 hybrid_clock.cc:648] HybridClock initialized: now 1790454067210030 us; error 29 us; skew 500 ppm
Sep 26 20:21:07 dist-test-slave-qsm9 krb5kdc[6535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454067, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:07.213019 7696 init.cc:377] Logged in from keytab as oryx/127.0.116.195@KRBTEST.COM (short username oryx)
I20260926 20:21:07.213871 7696 webserver.cc:460] Webserver started at http://127.0.116.195:35165/ using document root <none> and password file <none>
I20260926 20:21:07.214401 7696 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:07.214473 7696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:07.218474 7696 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:21:07.220713 7711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:07.222153 7696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:21:07.222280 7696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "65fda696e9f1430abd4889d32512860f"
format_stamp: "Formatted at 2026-09-26 20:21:05 on dist-test-slave-qsm9"
server_key: "bdfbd0fd632ff27bb02c53be687eb3a6"
server_key_iv: "10dc419e6fa9a79e698af21193428d2a"
server_key_version: "encryptionkey@0"
I20260926 20:21:07.222851 7696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:07.241703 7696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:07.245251 7696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:07.245473 7696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:07.246126 7696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:07.247045 7696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:07.247123 7696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:07.247197 7696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:07.247233 7696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:07.257539 7696 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:37643
I20260926 20:21:07.257563 7825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:37643 every 8 connection(s)
I20260926 20:21:07.258647 7696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:21:07.263952 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 7696
Sep 26 20:21:07 dist-test-slave-qsm9 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454067, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM
Sep 26 20:21:07 dist-test-slave-qsm9 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454067, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.252@KRBTEST.COM
Sep 26 20:21:07 dist-test-slave-qsm9 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454067, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.253@KRBTEST.COM
I20260926 20:21:07.277078 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.266432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43864 (local address 127.0.116.252:42181)
0926 20:21:07.267561 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:07.267565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:07.269188 (+ 1623us) server_negotiation.cc:407] Connection header received
0926 20:21:07.269380 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:07.269385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:07.269449 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:07.269533 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:07.270144 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.271045 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.273562 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.274345 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.274430 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.274611 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:07.276799 (+ 2188us) server_negotiation.cc:299] Negotiation successful
0926 20:21:07.276893 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1046,"thread_start_us":51,"threads_started":1}
I20260926 20:21:07.277809 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.266254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42181 (local address 127.0.0.1:43864)
0926 20:21:07.269107 (+ 2853us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:07.269122 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:07.269258 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:07.269673 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:07.269676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:07.269696 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:07.269794 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.269799 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.271191 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.271195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.272603 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.272617 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.274718 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.274722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.274845 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.274879 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 20:21:07.277524 (+ 2645us) client_negotiation.cc:240] Negotiation successful
0926 20:21:07.277633 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2787,"thread_start_us":62,"threads_started":1}
I20260926 20:21:07.278728 7381 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 20:21:07.279320 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.266207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43854 (local address 127.0.116.252:42181)
0926 20:21:07.267075 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:07.267081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:07.267304 (+ 223us) server_negotiation.cc:407] Connection header received
0926 20:21:07.267429 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:07.267435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:07.267524 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:07.267615 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:07.269626 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.270452 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.271527 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.271862 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.271968 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.274141 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:07.274148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:07.274154 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:07.274183 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:07.277261 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.277717 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.277723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.277728 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.277809 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.278125 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.278128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.278130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.278324 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:07.278427 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:07.279040 (+ 613us) server_negotiation.cc:299] Negotiation successful
0926 20:21:07.279173 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"spinlock_wait_cycles":896,"thread_start_us":84,"threads_started":1}
I20260926 20:21:07.279582 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.262159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:53899 (local address 127.0.116.253:36501)
0926 20:21:07.262287 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:07.262291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:07.262353 (+ 62us) server_negotiation.cc:407] Connection header received
0926 20:21:07.263086 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:07.263089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:07.263138 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:07.263212 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:07.264784 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.265612 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.267802 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.268139 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.268215 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.272375 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:07.272379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:07.272381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:07.272421 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:07.275658 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.277799 (+ 2141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.277803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.277805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.277900 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.278208 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.278211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.278215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.278436 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:07.278522 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:07.279261 (+ 739us) server_negotiation.cc:299] Negotiation successful
0926 20:21:07.279378 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:21:07.280121 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.260594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:43741 (local address 127.0.116.254:33869)
0926 20:21:07.260714 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:07.260719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:07.261631 (+ 912us) server_negotiation.cc:407] Connection header received
0926 20:21:07.262932 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:07.262935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:07.262983 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:07.263072 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:07.264791 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.265550 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.267551 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.267883 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.267970 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.272219 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:07.272223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:07.272226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:07.272252 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:07.275901 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.277180 (+ 1279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.277186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.277192 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.277306 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.277978 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.277981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.277982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.278200 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:07.278272 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:07.279843 (+ 1571us) server_negotiation.cc:299] Negotiation successful
0926 20:21:07.279962 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:21:07.280735 7845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72e44190f4b64472a35049cc412c4234. Latest consensus state: current_term: 1 leader_uuid: "72e44190f4b64472a35049cc412c4234" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: NON_VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: true } } }
I20260926 20:21:07.280871 7845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:21:07.281386 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.260946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.116.195:43741)
0926 20:21:07.261453 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:07.261504 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:07.262566 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:07.263134 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:07.263145 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:07.263833 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:07.264627 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.264638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.265739 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.265746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.267380 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.267392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.268145 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.268149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.268467 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.269302 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:07.269329 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:07.272033 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:07.276822 (+ 4789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:07.276827 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:07.276831 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:07.277057 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:07.277778 (+ 721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:07.277781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:07.277783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:07.277869 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:07.278359 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:07.278370 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:07.279632 (+ 1262us) client_negotiation.cc:769] Sending connection context
0926 20:21:07.280293 (+ 661us) client_negotiation.cc:240] Negotiation successful
0926 20:21:07.280578 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":313,"spinlock_wait_cycles":28800,"thread_start_us":559,"threads_started":1}
I20260926 20:21:07.281765 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.262005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:36501 (local address 127.0.116.195:53899)
0926 20:21:07.262282 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:07.262296 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:07.262566 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:07.263331 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:07.263336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:07.263821 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:07.264613 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.264633 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.265762 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.265765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.267689 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.267697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.268380 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.268383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.268470 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.270446 (+ 1976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:07.270461 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:07.272223 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:07.277382 (+ 5159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:07.277385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:07.277388 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:07.277683 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:07.278014 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:07.278017 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:07.278019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:07.278104 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:07.278578 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:07.278581 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:07.278974 (+ 393us) client_negotiation.cc:769] Sending connection context
0926 20:21:07.279771 (+ 797us) client_negotiation.cc:240] Negotiation successful
0926 20:21:07.280185 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":608,"spinlock_wait_cycles":15872,"thread_start_us":109,"threads_started":1}
I20260926 20:21:07.282985 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.260947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42181 (local address 127.0.116.195:59983)
0926 20:21:07.261771 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:07.261786 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:07.262567 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:07.263134 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:07.263143 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:07.263821 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:07.264615 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.264634 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.266487 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.266491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.267391 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.267403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.268146 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.268150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.268462 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.269689 (+ 1227us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:07.269704 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:07.272828 (+ 3124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:07.276209 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:07.276215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:07.276233 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:07.276577 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:07.276908 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:07.276911 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:07.276914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:07.277103 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:07.282048 (+ 4945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:07.282051 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:07.282154 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 20:21:07.282267 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 20:21:07.282420 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":891,"spinlock_wait_cycles":14464,"thread_start_us":155,"threads_started":1}
I20260926 20:21:07.283066 7827 heartbeater.cc:347] Connected to a master server at 127.0.116.254:33869
I20260926 20:21:07.283186 7556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.260533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:59983 (local address 127.0.116.252:42181)
0926 20:21:07.260673 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:07.260677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:07.261861 (+ 1184us) server_negotiation.cc:407] Connection header received
0926 20:21:07.262787 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:07.262791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:07.262845 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:07.262951 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:07.264798 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.265611 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.267547 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.267882 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.267948 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.274192 (+ 6244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:07.274195 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:07.274197 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:07.274216 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:07.276065 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.276698 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.276702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.276703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.276796 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.277210 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.277213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.277215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.277584 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:07.277703 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:07.282866 (+ 5163us) server_negotiation.cc:299] Negotiation successful
0926 20:21:07.283010 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:21:07.283329 7827 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:07.282960 7828 heartbeater.cc:347] Connected to a master server at 127.0.116.253:36501
I20260926 20:21:07.283640 7828 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:07.283803 7826 heartbeater.cc:347] Connected to a master server at 127.0.116.252:42181
I20260926 20:21:07.283907 7826 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:07.284076 7826 heartbeater.cc:511] Master 127.0.116.252:42181 requested a full tablet report, sending...
I20260926 20:21:07.284076 7827 heartbeater.cc:511] Master 127.0.116.254:33869 requested a full tablet report, sending...
I20260926 20:21:07.284077 7828 heartbeater.cc:511] Master 127.0.116.253:36501 requested a full tablet report, sending...
I20260926 20:21:07.285491 6581 ts_manager.cc:194] Re-registered known tserver with Master: 65fda696e9f1430abd4889d32512860f (127.0.116.195:37643)
I20260926 20:21:07.285499 7371 ts_manager.cc:194] Registered new tserver with Master: 65fda696e9f1430abd4889d32512860f (127.0.116.195:37643)
I20260926 20:21:07.285979 7188 ts_manager.cc:194] Registered new tserver with Master: 65fda696e9f1430abd4889d32512860f (127.0.116.195:37643)
I20260926 20:21:07.286311 6581 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:43741
I20260926 20:21:07.289719 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.267845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35278 (local address 127.0.116.254:33869)
0926 20:21:07.272817 (+ 4972us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:07.272823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:07.272839 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:21:07.273085 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:07.273088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:07.273143 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:07.273228 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:07.274721 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.275618 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.283190 (+ 7572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.283606 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.283715 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.285560 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:07.285564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:07.285567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:07.285594 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:07.287812 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.288354 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.288357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.288358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.288448 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.288744 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.288751 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.288755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.288979 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:07.289137 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:07.289381 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 20:21:07.289529 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":4899,"thread_start_us":102,"threads_started":1}
I20260926 20:21:07.290056 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.268526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49950 (local address 127.0.116.253:36501)
0926 20:21:07.270058 (+ 1532us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:07.270064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:07.273230 (+ 3166us) server_negotiation.cc:407] Connection header received
0926 20:21:07.273271 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:07.273274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:07.273330 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:07.273412 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:07.275790 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.276631 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.280823 (+ 4192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.281195 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.281321 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.284551 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:07.284556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:07.284559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:07.284588 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:07.288159 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.288592 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.288596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.288598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.288716 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.289080 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.289083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.289085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.289329 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:07.289472 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:07.289722 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 20:21:07.289868 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1439,"thread_start_us":72,"threads_started":1}
I20260926 20:21:07.292774 6581 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:21:07.296065 6581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:21:07.298606 7145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72e44190f4b64472a35049cc412c4234" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: NON_VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: true } } }
I20260926 20:21:07.298765 7145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:07.302480 7145 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234: attempting to promote NON_VOTER 8a4b410506c6418e859f15f84740f6d7 to VOTER
I20260926 20:21:07.302820 7145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: false } }
I20260926 20:21:07.304005 7380 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [term 1 LEARNER]: Refusing update from remote peer 72e44190f4b64472a35049cc412c4234: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 20:21:07.304009 7198 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 1 FOLLOWER]: Refusing update from remote peer 72e44190f4b64472a35049cc412c4234: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 20:21:07.304376 7145 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 20:21:07.304668 7145 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 20:21:07.306334 7847 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8a4b410506c6418e859f15f84740f6d7 (127.0.116.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: false } } }
I20260926 20:21:07.307215 7198 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8a4b410506c6418e859f15f84740f6d7 (127.0.116.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: false } } }
I20260926 20:21:07.308048 7847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "72e44190f4b64472a35049cc412c4234" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: false } } }
I20260926 20:21:07.308142 7847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:07.308300 7847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72e44190f4b64472a35049cc412c4234" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: false } } }
I20260926 20:21:07.308377 7847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:07.308466 7847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72e44190f4b64472a35049cc412c4234" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: false } } }
I20260926 20:21:07.308562 7847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:07.308239 7381 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8a4b410506c6418e859f15f84740f6d7 (127.0.116.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: false } } }
I20260926 20:21:07.309949 7845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "72e44190f4b64472a35049cc412c4234" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: false } } }
I20260926 20:21:07.310062 7845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:21:07.309605 7857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "72e44190f4b64472a35049cc412c4234" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: false } } }
I20260926 20:21:07.310457 7857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:21:07.318781 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.312674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:37643 (local address 127.0.0.1:60546)
0926 20:21:07.313167 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:07.313183 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:07.313263 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:07.314514 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:07.314517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:07.314534 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:07.314617 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.314622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.316103 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.316106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.317381 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.317392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.318383 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.318386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.318469 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.318489 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 20:21:07.318547 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 20:21:07.318623 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":65}
I20260926 20:21:07.318823 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.312853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60546 (local address 127.0.116.195:37643)
0926 20:21:07.313931 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:07.313941 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:07.313959 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:21:07.314027 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:07.314031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:07.314207 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:07.314371 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:07.314870 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.315982 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.317505 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.318257 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.318373 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.318520 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:07.318585 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 20:21:07.318680 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":981,"thread_start_us":112,"threads_started":1}
I20260926 20:21:07.318944 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.313439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60000 (local address 127.0.116.193:35253)
0926 20:21:07.313945 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:07.313957 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:07.313976 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:21:07.314036 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:07.314039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:07.314207 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:07.314329 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:07.314756 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.315997 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.317150 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.317875 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.317993 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.318138 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:07.318702 (+ 564us) server_negotiation.cc:299] Negotiation successful
0926 20:21:07.318794 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":75,"threads_started":1}
I20260926 20:21:07.319592 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.313079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:41615 (local address 127.0.0.1:36476)
0926 20:21:07.313645 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:07.313658 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:07.313746 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:07.315264 (+ 1518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:07.315267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:07.315284 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:07.315375 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.315392 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.316742 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.316746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.317777 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.317784 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.319184 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.319191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.319281 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.319302 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 20:21:07.319358 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 20:21:07.319426 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":21,"thread_start_us":80,"threads_started":1}
I20260926 20:21:07.319624 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.313433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.0.116.194:41615)
0926 20:21:07.314603 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:07.314614 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:07.314634 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:21:07.314704 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:07.314711 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:07.314908 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:07.315569 (+ 661us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:07.315684 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.316618 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.317883 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.319077 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.319187 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.319328 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:07.319391 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 20:21:07.319472 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1054,"thread_start_us":93,"threads_started":1}
I20260926 20:21:07.320024 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.312884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:35253 (local address 127.0.0.1:60000)
0926 20:21:07.313606 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:07.313619 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:07.313706 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:07.314340 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:07.314343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:07.314359 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:07.314497 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.314506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.316130 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.316136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.317029 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.317038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.318314 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.318317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.318413 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.318440 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 20:21:07.318515 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:21:07.318593 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":36,"thread_start_us":120,"threads_started":1}
I20260926 20:21:07.322316 7484 tablet_service.cc:1467] Processing CreateTablet for tablet d8b2577e21034cae87edc7432970ab2b (DEFAULT_TABLE table=test-table [id=70813ec3b22d47b09f962f317561435c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:21:07.322263 7760 tablet_service.cc:1467] Processing CreateTablet for tablet d8b2577e21034cae87edc7432970ab2b (DEFAULT_TABLE table=test-table [id=70813ec3b22d47b09f962f317561435c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:21:07.322261 7623 tablet_service.cc:1467] Processing CreateTablet for tablet d8b2577e21034cae87edc7432970ab2b (DEFAULT_TABLE table=test-table [id=70813ec3b22d47b09f962f317561435c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:21:07.323285 7484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8b2577e21034cae87edc7432970ab2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:07.323302 7623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8b2577e21034cae87edc7432970ab2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:07.323905 7760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8b2577e21034cae87edc7432970ab2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:07.332178 7863 tablet_bootstrap.cc:492] T d8b2577e21034cae87edc7432970ab2b P dbabb7ce4283472aa26a1dc3d52ee4fb: Bootstrap starting.
I20260926 20:21:07.332473 7865 tablet_bootstrap.cc:492] T d8b2577e21034cae87edc7432970ab2b P 8a913df7cca14f21980b78e45afb13ed: Bootstrap starting.
I20260926 20:21:07.334414 7865 tablet_bootstrap.cc:654] T d8b2577e21034cae87edc7432970ab2b P 8a913df7cca14f21980b78e45afb13ed: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:07.334460 7863 tablet_bootstrap.cc:654] T d8b2577e21034cae87edc7432970ab2b P dbabb7ce4283472aa26a1dc3d52ee4fb: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:07.335163 7865 log.cc:824] T d8b2577e21034cae87edc7432970ab2b P 8a913df7cca14f21980b78e45afb13ed: Log is configured to *not* fsync() on all Append() calls
I20260926 20:21:07.335207 7863 log.cc:824] T d8b2577e21034cae87edc7432970ab2b P dbabb7ce4283472aa26a1dc3d52ee4fb: Log is configured to *not* fsync() on all Append() calls
I20260926 20:21:07.335871 7864 tablet_bootstrap.cc:492] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f: Bootstrap starting.
I20260926 20:21:07.337229 7865 tablet_bootstrap.cc:492] T d8b2577e21034cae87edc7432970ab2b P 8a913df7cca14f21980b78e45afb13ed: No bootstrap required, opened a new log
I20260926 20:21:07.337231 7863 tablet_bootstrap.cc:492] T d8b2577e21034cae87edc7432970ab2b P dbabb7ce4283472aa26a1dc3d52ee4fb: No bootstrap required, opened a new log
I20260926 20:21:07.337412 7865 ts_tablet_manager.cc:1397] T d8b2577e21034cae87edc7432970ab2b P 8a913df7cca14f21980b78e45afb13ed: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260926 20:21:07.337411 7863 ts_tablet_manager.cc:1397] T d8b2577e21034cae87edc7432970ab2b P dbabb7ce4283472aa26a1dc3d52ee4fb: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260926 20:21:07.339318 7864 tablet_bootstrap.cc:654] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:07.340263 7864 log.cc:824] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:21:07.341846 7865 raft_consensus.cc:348] T d8b2577e21034cae87edc7432970ab2b P 8a913df7cca14f21980b78e45afb13ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fda696e9f1430abd4889d32512860f" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 37643 } } peers { permanent_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35253 } } peers { permanent_uuid: "8a913df7cca14f21980b78e45afb13ed" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41615 } }
I20260926 20:21:07.341845 7863 raft_consensus.cc:348] T d8b2577e21034cae87edc7432970ab2b P dbabb7ce4283472aa26a1dc3d52ee4fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fda696e9f1430abd4889d32512860f" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 37643 } } peers { permanent_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35253 } } peers { permanent_uuid: "8a913df7cca14f21980b78e45afb13ed" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41615 } }
I20260926 20:21:07.342110 7865 raft_consensus.cc:374] T d8b2577e21034cae87edc7432970ab2b P 8a913df7cca14f21980b78e45afb13ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:07.342108 7863 raft_consensus.cc:374] T d8b2577e21034cae87edc7432970ab2b P dbabb7ce4283472aa26a1dc3d52ee4fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:07.342177 7865 raft_consensus.cc:729] T d8b2577e21034cae87edc7432970ab2b P 8a913df7cca14f21980b78e45afb13ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a913df7cca14f21980b78e45afb13ed, State: Initialized, Role: FOLLOWER
I20260926 20:21:07.342185 7863 raft_consensus.cc:729] T d8b2577e21034cae87edc7432970ab2b P dbabb7ce4283472aa26a1dc3d52ee4fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabb7ce4283472aa26a1dc3d52ee4fb, State: Initialized, Role: FOLLOWER
I20260926 20:21:07.342636 7865 consensus_queue.cc:260] T d8b2577e21034cae87edc7432970ab2b P 8a913df7cca14f21980b78e45afb13ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fda696e9f1430abd4889d32512860f" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 37643 } } peers { permanent_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35253 } } peers { permanent_uuid: "8a913df7cca14f21980b78e45afb13ed" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41615 } }
I20260926 20:21:07.342661 7863 consensus_queue.cc:260] T d8b2577e21034cae87edc7432970ab2b P dbabb7ce4283472aa26a1dc3d52ee4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fda696e9f1430abd4889d32512860f" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 37643 } } peers { permanent_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35253 } } peers { permanent_uuid: "8a913df7cca14f21980b78e45afb13ed" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41615 } }
I20260926 20:21:07.343708 7865 ts_tablet_manager.cc:1428] T d8b2577e21034cae87edc7432970ab2b P 8a913df7cca14f21980b78e45afb13ed: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260926 20:21:07.343783 7863 ts_tablet_manager.cc:1428] T d8b2577e21034cae87edc7432970ab2b P dbabb7ce4283472aa26a1dc3d52ee4fb: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260926 20:21:07.343963 7552 heartbeater.cc:503] Master 127.0.116.254:33869 was elected leader, sending a full tablet report...
I20260926 20:21:07.346611 7864 tablet_bootstrap.cc:492] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f: No bootstrap required, opened a new log
I20260926 20:21:07.346637 7690 heartbeater.cc:503] Master 127.0.116.254:33869 was elected leader, sending a full tablet report...
I20260926 20:21:07.347198 7864 ts_tablet_manager.cc:1397] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260926 20:21:07.352097 7864 raft_consensus.cc:348] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fda696e9f1430abd4889d32512860f" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 37643 } } peers { permanent_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35253 } } peers { permanent_uuid: "8a913df7cca14f21980b78e45afb13ed" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41615 } }
I20260926 20:21:07.352461 7864 raft_consensus.cc:374] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:07.352531 7864 raft_consensus.cc:729] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65fda696e9f1430abd4889d32512860f, State: Initialized, Role: FOLLOWER
I20260926 20:21:07.353106 7864 consensus_queue.cc:260] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fda696e9f1430abd4889d32512860f" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 37643 } } peers { permanent_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35253 } } peers { permanent_uuid: "8a913df7cca14f21980b78e45afb13ed" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41615 } }
I20260926 20:21:07.353920 7827 heartbeater.cc:503] Master 127.0.116.254:33869 was elected leader, sending a full tablet report...
I20260926 20:21:07.354231 7864 ts_tablet_manager.cc:1428] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 20:21:07.493870 7871 raft_consensus.cc:482] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:21:07.494117 7871 raft_consensus.cc:504] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fda696e9f1430abd4889d32512860f" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 37643 } } peers { permanent_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35253 } } peers { permanent_uuid: "8a913df7cca14f21980b78e45afb13ed" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41615 } }
I20260926 20:21:07.495457 7871 leader_election.cc:288] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbabb7ce4283472aa26a1dc3d52ee4fb (127.0.116.193:35253), 8a913df7cca14f21980b78e45afb13ed (127.0.116.194:41615)
I20260926 20:21:07.500795 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.495711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:35253 (local address 127.0.116.195:47839)
0926 20:21:07.495957 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:07.495985 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:07.496086 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:07.496370 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:07.496373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:07.496400 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:07.496616 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.496621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.497693 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.497697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.499258 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.499270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.500334 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.500338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.500459 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.500489 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 20:21:07.500564 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:21:07.500630 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":183,"spinlock_wait_cycles":19456}
I20260926 20:21:07.500849 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.495812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:47839 (local address 127.0.116.193:35253)
0926 20:21:07.496058 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:07.496063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:07.496083 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:21:07.496203 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:07.496206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:07.496255 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:07.496342 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:07.496763 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.497576 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.499421 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.500198 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.500339 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.500397 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:07.500611 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 20:21:07.500706 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260926 20:21:07.500996 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.495889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:48941 (local address 127.0.116.194:41615)
0926 20:21:07.496114 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:07.496120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:07.496133 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:21:07.496203 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:07.496206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:07.496255 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:07.496342 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:07.496763 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.497564 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.499421 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.500191 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.500339 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.500397 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:07.500785 (+ 388us) server_negotiation.cc:299] Negotiation successful
0926 20:21:07.500877 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260926 20:21:07.501015 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.495819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:41615 (local address 127.0.116.195:48941)
0926 20:21:07.495962 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:07.495975 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:07.496086 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:07.496370 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:07.496372 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:07.496398 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:07.496612 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.496621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.497693 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.497697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.499260 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:07.499270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.500468 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.500472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:07.500644 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.500667 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:21:07.500751 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:21:07.500824 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":161,"spinlock_wait_cycles":15616}
I20260926 20:21:07.501776 7643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8b2577e21034cae87edc7432970ab2b" candidate_uuid: "65fda696e9f1430abd4889d32512860f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a913df7cca14f21980b78e45afb13ed" is_pre_election: true
I20260926 20:21:07.501770 7504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8b2577e21034cae87edc7432970ab2b" candidate_uuid: "65fda696e9f1430abd4889d32512860f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" is_pre_election: true
I20260926 20:21:07.502116 7504 raft_consensus.cc:2454] T d8b2577e21034cae87edc7432970ab2b P dbabb7ce4283472aa26a1dc3d52ee4fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65fda696e9f1430abd4889d32512860f in term 0.
I20260926 20:21:07.502112 7643 raft_consensus.cc:2454] T d8b2577e21034cae87edc7432970ab2b P 8a913df7cca14f21980b78e45afb13ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65fda696e9f1430abd4889d32512860f in term 0.
I20260926 20:21:07.502637 7713 leader_election.cc:302] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65fda696e9f1430abd4889d32512860f, 8a913df7cca14f21980b78e45afb13ed; no voters:
I20260926 20:21:07.502895 7871 raft_consensus.cc:2781] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:21:07.502982 7871 raft_consensus.cc:482] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:21:07.503031 7871 raft_consensus.cc:3037] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:07.504210 7871 raft_consensus.cc:504] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fda696e9f1430abd4889d32512860f" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 37643 } } peers { permanent_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35253 } } peers { permanent_uuid: "8a913df7cca14f21980b78e45afb13ed" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41615 } }
I20260926 20:21:07.504617 7871 leader_election.cc:288] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f [CANDIDATE]: Term 1 election: Requested vote from peers dbabb7ce4283472aa26a1dc3d52ee4fb (127.0.116.193:35253), 8a913df7cca14f21980b78e45afb13ed (127.0.116.194:41615)
I20260926 20:21:07.505002 7643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8b2577e21034cae87edc7432970ab2b" candidate_uuid: "65fda696e9f1430abd4889d32512860f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a913df7cca14f21980b78e45afb13ed"
I20260926 20:21:07.505002 7504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8b2577e21034cae87edc7432970ab2b" candidate_uuid: "65fda696e9f1430abd4889d32512860f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb"
I20260926 20:21:07.505167 7643 raft_consensus.cc:3037] T d8b2577e21034cae87edc7432970ab2b P 8a913df7cca14f21980b78e45afb13ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:07.505160 7504 raft_consensus.cc:3037] T d8b2577e21034cae87edc7432970ab2b P dbabb7ce4283472aa26a1dc3d52ee4fb [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:07.506508 7643 raft_consensus.cc:2454] T d8b2577e21034cae87edc7432970ab2b P 8a913df7cca14f21980b78e45afb13ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65fda696e9f1430abd4889d32512860f in term 1.
I20260926 20:21:07.506508 7504 raft_consensus.cc:2454] T d8b2577e21034cae87edc7432970ab2b P dbabb7ce4283472aa26a1dc3d52ee4fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65fda696e9f1430abd4889d32512860f in term 1.
I20260926 20:21:07.506933 7715 leader_election.cc:302] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 65fda696e9f1430abd4889d32512860f, 8a913df7cca14f21980b78e45afb13ed, dbabb7ce4283472aa26a1dc3d52ee4fb; no voters:
I20260926 20:21:07.507153 7871 raft_consensus.cc:2781] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:21:07.507393 7871 raft_consensus.cc:686] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f [term 1 LEADER]: Becoming Leader. State: Replica: 65fda696e9f1430abd4889d32512860f, State: Running, Role: LEADER
I20260926 20:21:07.507903 7871 consensus_queue.cc:237] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fda696e9f1430abd4889d32512860f" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 37643 } } peers { permanent_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35253 } } peers { permanent_uuid: "8a913df7cca14f21980b78e45afb13ed" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41615 } }
W20260926 20:21:07.510036 7829 tablet.cc:2369] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:21:07.512116 6580 catalog_manager.cc:5495] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f reported cstate change: term changed from 0 to 1, leader changed from <none> to 65fda696e9f1430abd4889d32512860f (127.0.116.195). New cstate: current_term: 1 leader_uuid: "65fda696e9f1430abd4889d32512860f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fda696e9f1430abd4889d32512860f" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 37643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a913df7cca14f21980b78e45afb13ed" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41615 } health_report { overall_health: UNKNOWN } } }
W20260926 20:21:07.556778 7692 tablet.cc:2369] T d8b2577e21034cae87edc7432970ab2b P 8a913df7cca14f21980b78e45afb13ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:21:07.561429 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.557333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.0.116.195:37643)
0926 20:21:07.557517 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:07.557521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:07.557535 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:21:07.557638 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:07.557640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:07.557695 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:07.557785 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:21:07.558113 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.558620 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.559767 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.560066 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.560167 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.560510 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:21:07.560980 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:21:07.561088 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:07.561226 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 20:21:07.561285 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 20:21:07.568639 7643 raft_consensus.cc:1261] T d8b2577e21034cae87edc7432970ab2b P 8a913df7cca14f21980b78e45afb13ed [term 1 FOLLOWER]: Refusing update from remote peer 65fda696e9f1430abd4889d32512860f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:21:07.568693 7504 raft_consensus.cc:1261] T d8b2577e21034cae87edc7432970ab2b P dbabb7ce4283472aa26a1dc3d52ee4fb [term 1 FOLLOWER]: Refusing update from remote peer 65fda696e9f1430abd4889d32512860f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:21:07.569571 7872 consensus_queue.cc:1035] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a913df7cca14f21980b78e45afb13ed" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:21:07.569788 7871 consensus_queue.cc:1035] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:21:07.580529 7876 mvcc.cc:205] Tried to move back new op lower bound from 7333699860751568896 to 7333699860515463168. Current Snapshot: MvccSnapshot[applied={T|T < 7333699860751568896}]
I20260926 20:21:07.590251 6581 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:21:07.590420 6581 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:21:07.594913 6581 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72e44190f4b64472a35049cc412c4234: Sending DeleteTablet for 3 replicas of tablet d8b2577e21034cae87edc7432970ab2b
I20260926 20:21:07.595752 7484 tablet_service.cc:1514] Processing DeleteTablet for tablet d8b2577e21034cae87edc7432970ab2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:07 UTC) from {username='oryx', principal='oryx/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:60000
I20260926 20:21:07.595781 7760 tablet_service.cc:1514] Processing DeleteTablet for tablet d8b2577e21034cae87edc7432970ab2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:07 UTC) from {username='oryx', principal='oryx/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:60546
I20260926 20:21:07.595818 7623 tablet_service.cc:1514] Processing DeleteTablet for tablet d8b2577e21034cae87edc7432970ab2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:07 UTC) from {username='oryx', principal='oryx/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:36476
I20260926 20:21:07.596378 7888 tablet_replica.cc:330] T d8b2577e21034cae87edc7432970ab2b P dbabb7ce4283472aa26a1dc3d52ee4fb: stopping tablet replica
I20260926 20:21:07.596727 7888 raft_consensus.cc:2229] T d8b2577e21034cae87edc7432970ab2b P dbabb7ce4283472aa26a1dc3d52ee4fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:21:07.597116 7888 raft_consensus.cc:2258] T d8b2577e21034cae87edc7432970ab2b P dbabb7ce4283472aa26a1dc3d52ee4fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:21:07.597474 7886 tablet_replica.cc:330] T d8b2577e21034cae87edc7432970ab2b P 8a913df7cca14f21980b78e45afb13ed: stopping tablet replica
I20260926 20:21:07.597790 7886 raft_consensus.cc:2229] T d8b2577e21034cae87edc7432970ab2b P 8a913df7cca14f21980b78e45afb13ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:21:07.598124 7886 raft_consensus.cc:2258] T d8b2577e21034cae87edc7432970ab2b P 8a913df7cca14f21980b78e45afb13ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:21:07.598134 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 6548
I20260926 20:21:07.600406 7888 ts_tablet_manager.cc:1905] T d8b2577e21034cae87edc7432970ab2b P dbabb7ce4283472aa26a1dc3d52ee4fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:21:07.604298 7888 ts_tablet_manager.cc:1918] T d8b2577e21034cae87edc7432970ab2b P dbabb7ce4283472aa26a1dc3d52ee4fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:21:07.604419 7888 log.cc:1196] T d8b2577e21034cae87edc7432970ab2b P dbabb7ce4283472aa26a1dc3d52ee4fb: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-0/wal/wals/d8b2577e21034cae87edc7432970ab2b
I20260926 20:21:07.605048 7887 tablet_replica.cc:330] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f: stopping tablet replica
I20260926 20:21:07.605417 7887 raft_consensus.cc:2229] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:21:07.605839 7887 raft_consensus.cc:2258] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:21:07.607887 7886 ts_tablet_manager.cc:1905] T d8b2577e21034cae87edc7432970ab2b P 8a913df7cca14f21980b78e45afb13ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:21:07.610671 7886 ts_tablet_manager.cc:1918] T d8b2577e21034cae87edc7432970ab2b P 8a913df7cca14f21980b78e45afb13ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:21:07.612505 7887 ts_tablet_manager.cc:1905] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:21:07.614640 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.597989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49954 (local address 127.0.116.253:36501)
0926 20:21:07.598086 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:07.598090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:07.604055 (+ 5965us) server_negotiation.cc:407] Connection header received
0926 20:21:07.604268 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:07.604272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:07.604326 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:07.604379 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:07.605730 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.606553 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.607890 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.608270 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.608344 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.610888 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:07.610892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:07.610895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:07.610924 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:07.612726 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.613296 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.613300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.613303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.613387 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.613672 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.613675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.613677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.613883 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:07.613979 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:07.614335 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 20:21:07.614472 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:21:07.615299 7887 ts_tablet_manager.cc:1918] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:21:07.615406 7887 log.cc:1196] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/d8b2577e21034cae87edc7432970ab2b
I20260926 20:21:07.615787 7887 ts_tablet_manager.cc:1939] T d8b2577e21034cae87edc7432970ab2b P 65fda696e9f1430abd4889d32512860f: Deleting consensus metadata
I20260926 20:21:07.610770 7886 log.cc:1196] T d8b2577e21034cae87edc7432970ab2b P 8a913df7cca14f21980b78e45afb13ed: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/d8b2577e21034cae87edc7432970ab2b
W20260926 20:21:07.617003 7712 connection.cc:441] server connection from 127.0.0.1:60546 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60546 (request call id 1) could send its response
I20260926 20:21:07.617442 7886 ts_tablet_manager.cc:1939] T d8b2577e21034cae87edc7432970ab2b P 8a913df7cca14f21980b78e45afb13ed: Deleting consensus metadata
I20260926 20:21:07.604748 7888 ts_tablet_manager.cc:1939] T d8b2577e21034cae87edc7432970ab2b P dbabb7ce4283472aa26a1dc3d52ee4fb: Deleting consensus metadata
W20260926 20:21:07.618741 7575 connection.cc:441] server connection from 127.0.0.1:36476 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36476 (request call id 1) could send its response
W20260926 20:21:07.619829 7440 connection.cc:441] server connection from 127.0.0.1:60000 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60000 (request call id 1) could send its response
I20260926 20:21:07.623051 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.610764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43872 (local address 127.0.116.252:42181)
0926 20:21:07.610884 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:07.610888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:07.611468 (+ 580us) server_negotiation.cc:407] Connection header received
0926 20:21:07.611693 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:07.611697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:07.611751 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:07.611804 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:07.612674 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.613441 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.615944 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.616264 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.616343 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.618282 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:07.618286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:07.618289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:07.618315 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:07.621000 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.621497 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.621501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.621502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.621587 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.621868 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.621871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.621873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.622072 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:07.622185 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:07.622769 (+ 584us) server_negotiation.cc:299] Negotiation successful
0926 20:21:07.622889 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:21:07.627092 7556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.597763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43870 (local address 127.0.116.252:42181)
0926 20:21:07.597913 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:07.597917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:07.616491 (+ 18574us) server_negotiation.cc:407] Connection header received
0926 20:21:07.616658 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:07.616661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:07.616709 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:07.617189 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:07.618165 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.619846 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.621235 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.621571 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.621642 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.623437 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:07.623441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:07.623444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:07.623468 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:07.625140 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.625619 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.625623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.625624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.625706 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.626037 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.626040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.626042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.626271 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:07.626417 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:07.626822 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 20:21:07.626930 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:21:07.629015 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:07.615120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49970 (local address 127.0.116.253:36501)
0926 20:21:07.615258 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:07.615262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:07.620189 (+ 4927us) server_negotiation.cc:407] Connection header received
0926 20:21:07.620389 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:07.620392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:07.620466 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:07.620521 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:07.621490 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.622329 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.623437 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:07.623828 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:07.623913 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:07.625527 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:07.625530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:07.625533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:07.625565 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:07.627227 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.627731 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.627735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.627737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.627820 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:07.628208 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:07.628211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:07.628213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:07.628424 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:07.628547 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:07.628740 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 20:21:07.628854 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 20:21:08.584347 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:08.583827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.116.193:56155)
0926 20:21:08.584133 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:08.584241 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":114,"threads_started":1}
W20260926 20:21:08.585057 7552 heartbeater.cc:644] Failed to heartbeat to 127.0.116.254:33869 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
I20260926 20:21:08.585431 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:08.584876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.116.194:44113)
0926 20:21:08.585230 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:08.585323 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":71,"threads_started":1}
I20260926 20:21:08.585860 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:08.585270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.116.195:38379)
0926 20:21:08.585676 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:08.585764 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":66,"threads_started":1}
W20260926 20:21:08.585939 7690 heartbeater.cc:644] Failed to heartbeat to 127.0.116.254:33869 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
W20260926 20:21:08.586184 7827 heartbeater.cc:644] Failed to heartbeat to 127.0.116.254:33869 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
I20260926 20:21:09.095463 7900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 72e44190f4b64472a35049cc412c4234)
I20260926 20:21:09.095680 7900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: false } }
I20260926 20:21:09.095821 7901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 72e44190f4b64472a35049cc412c4234)
I20260926 20:21:09.096056 7901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: false } }
I20260926 20:21:09.096334 7900 leader_election.cc:288] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72e44190f4b64472a35049cc412c4234 (127.0.116.254:33869), 8a4b410506c6418e859f15f84740f6d7 (127.0.116.252:42181)
I20260926 20:21:09.097330 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.096594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.0.1:35724)
0926 20:21:09.097124 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.097218 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":75,"threads_started":1}
W20260926 20:21:09.097640 7172 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
I20260926 20:21:09.098625 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.098374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.0.1:35750)
0926 20:21:09.098502 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.098550 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260926 20:21:09.098961 7172 leader_election.cc:334] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 72e44190f4b64472a35049cc412c4234 (127.0.116.254:33869): Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
I20260926 20:21:09.099345 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.097975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.0.1:35738)
0926 20:21:09.098977 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.099049 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":926,"thread_start_us":72,"threads_started":1}
W20260926 20:21:09.099572 7355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
I20260926 20:21:09.099169 7901 leader_election.cc:288] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72e44190f4b64472a35049cc412c4234 (127.0.116.254:33869), 00e0d5ec1e01475a871d550cf2f981e0 (127.0.116.253:36501)
I20260926 20:21:09.100857 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.100506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.0.1:35758)
0926 20:21:09.100712 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.100770 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260926 20:21:09.101001 7355 leader_election.cc:334] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 72e44190f4b64472a35049cc412c4234 (127.0.116.254:33869): Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
I20260926 20:21:09.101892 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.096718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43888 (local address 127.0.116.252:42181)
0926 20:21:09.096972 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:09.096978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:09.097419 (+ 441us) server_negotiation.cc:407] Connection header received
0926 20:21:09.097475 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:09.097478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:09.097534 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:09.097608 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:09.097994 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:09.098793 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:09.100544 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:09.101185 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:09.101280 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:09.101342 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:09.101666 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 20:21:09.101737 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260926 20:21:09.101941 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.096593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42181 (local address 127.0.0.1:43888)
0926 20:21:09.097187 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.097201 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:09.097316 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:09.097730 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:09.097733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:09.097752 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:09.097857 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:09.097867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:09.098989 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:09.098992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:09.100422 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:09.100434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:09.101314 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:09.101320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:09.101477 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:09.101519 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 20:21:09.101641 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 20:21:09.101747 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":19,"thread_start_us":69,"threads_started":1}
I20260926 20:21:09.102706 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00e0d5ec1e01475a871d550cf2f981e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8a4b410506c6418e859f15f84740f6d7" is_pre_election: true
I20260926 20:21:09.102947 7381 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00e0d5ec1e01475a871d550cf2f981e0 in term 1.
I20260926 20:21:09.103317 7175 leader_election.cc:302] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00e0d5ec1e01475a871d550cf2f981e0, 8a4b410506c6418e859f15f84740f6d7; no voters: 72e44190f4b64472a35049cc412c4234
I20260926 20:21:09.103617 7900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 20:21:09.103689 7900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 72e44190f4b64472a35049cc412c4234)
I20260926 20:21:09.103740 7900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:21:09.104521 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.098742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49986 (local address 127.0.116.253:36501)
0926 20:21:09.099342 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:09.099351 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:09.099718 (+ 367us) server_negotiation.cc:407] Connection header received
0926 20:21:09.099906 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:09.099909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:09.099954 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:09.100007 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:09.100450 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:09.101356 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:09.102857 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:09.103709 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:09.103832 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:09.103921 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:09.104324 (+ 403us) server_negotiation.cc:299] Negotiation successful
0926 20:21:09.104406 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":503,"thread_start_us":110,"threads_started":1}
I20260926 20:21:09.104599 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.098451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:36501 (local address 127.0.0.1:49986)
0926 20:21:09.099593 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.099607 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:09.099788 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:09.100235 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:09.100238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:09.100253 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:09.100333 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:09.100339 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:09.101440 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:09.101445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:09.102672 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:09.102682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:09.103988 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:09.103994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:09.104124 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:09.104162 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 20:21:09.104248 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 20:21:09.104349 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"thread_start_us":79,"threads_started":1}
I20260926 20:21:09.105149 7900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: false } }
I20260926 20:21:09.105206 7198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a4b410506c6418e859f15f84740f6d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "00e0d5ec1e01475a871d550cf2f981e0" is_pre_election: true
I20260926 20:21:09.105485 7198 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a4b410506c6418e859f15f84740f6d7 in current term 2: Already voted for candidate 00e0d5ec1e01475a871d550cf2f981e0 in this term.
I20260926 20:21:09.105607 7900 leader_election.cc:288] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [CANDIDATE]: Term 2 election: Requested vote from peers 72e44190f4b64472a35049cc412c4234 (127.0.116.254:33869), 8a4b410506c6418e859f15f84740f6d7 (127.0.116.252:42181)
I20260926 20:21:09.106010 7355 leader_election.cc:302] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a4b410506c6418e859f15f84740f6d7; no voters: 00e0d5ec1e01475a871d550cf2f981e0, 72e44190f4b64472a35049cc412c4234
I20260926 20:21:09.106065 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.105711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.0.1:35770)
0926 20:21:09.105858 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.105983 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 20:21:09.106017 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00e0d5ec1e01475a871d550cf2f981e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8a4b410506c6418e859f15f84740f6d7"
I20260926 20:21:09.106149 7381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:21:09.106992 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.106713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.0.1:35784)
0926 20:21:09.106845 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.106912 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 20:21:09.107087 7172 leader_election.cc:334] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 72e44190f4b64472a35049cc412c4234 (127.0.116.254:33869): Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
I20260926 20:21:09.107393 7381 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00e0d5ec1e01475a871d550cf2f981e0 in term 2.
I20260926 20:21:09.107550 7901 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:21:09.107786 7175 leader_election.cc:302] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00e0d5ec1e01475a871d550cf2f981e0, 8a4b410506c6418e859f15f84740f6d7; no voters: 72e44190f4b64472a35049cc412c4234
I20260926 20:21:09.107955 7900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:21:09.108183 7900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [term 2 LEADER]: Becoming Leader. State: Replica: 00e0d5ec1e01475a871d550cf2f981e0, State: Running, Role: LEADER
I20260926 20:21:09.108480 7900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: false } }
I20260926 20:21:09.109676 7910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00e0d5ec1e01475a871d550cf2f981e0. Latest consensus state: current_term: 2 leader_uuid: "00e0d5ec1e01475a871d550cf2f981e0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: false } } }
I20260926 20:21:09.109797 7910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:09.110085 7912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:21:09.112462 7912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:21:09.112910 7912 catalog_manager.cc:1259] Loaded cluster ID: a236558215094ba299536d94a3af8346
I20260926 20:21:09.112978 7912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:21:09.113531 7912 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:21:09.113914 7912 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0: Loaded TSK: 0
I20260926 20:21:09.114614 7912 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:21:09.151661 7187 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:21:09.154875 7187 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:21:09.158159 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.157857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.0.1:35790)
0926 20:21:09.157980 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.158069 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 20:21:09.158282 7381 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [term 2 FOLLOWER]: Refusing update from remote peer 00e0d5ec1e01475a871d550cf2f981e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 20:21:09.158808 7910 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 20:21:09.158900 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.158669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.0.1:35796)
0926 20:21:09.158789 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.158844 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 20:21:09.159049 7172 consensus_peers.cc:487] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 -> Peer 72e44190f4b64472a35049cc412c4234 (127.0.116.254:33869): Couldn't send request to peer 72e44190f4b64472a35049cc412c4234. Status: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 20:21:09.160463 7901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00e0d5ec1e01475a871d550cf2f981e0. Latest consensus state: current_term: 2 leader_uuid: "00e0d5ec1e01475a871d550cf2f981e0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: false } } }
I20260926 20:21:09.160598 7901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:21:09.161509 7913 mvcc.cc:205] Tried to move back new op lower bound from 7333699867266859008 to 7333699867071160320. Current Snapshot: MvccSnapshot[applied={T|T < 7333699867266859008}]
I20260926 20:21:09.161722 7910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "00e0d5ec1e01475a871d550cf2f981e0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: false } } }
I20260926 20:21:09.161844 7910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:09.162005 7914 mvcc.cc:205] Tried to move back new op lower bound from 7333699867266859008 to 7333699867071160320. Current Snapshot: MvccSnapshot[applied={T|T < 7333699867266859008}]
I20260926 20:21:09.162410 7901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "00e0d5ec1e01475a871d550cf2f981e0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "72e44190f4b64472a35049cc412c4234" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33869 } } peers { permanent_uuid: "00e0d5ec1e01475a871d550cf2f981e0" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 36501 } attrs { promote: false } } peers { permanent_uuid: "8a4b410506c6418e859f15f84740f6d7" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42181 } attrs { promote: false } } }
I20260926 20:21:09.162526 7901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4b410506c6418e859f15f84740f6d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:21:09.171775 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.167862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:41615 (local address 127.0.0.1:36482)
0926 20:21:09.168022 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.168038 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:09.168149 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:09.168862 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:09.168866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:09.168883 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:09.169007 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:09.169015 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:09.169886 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:09.169893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:09.170623 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:09.170630 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:09.171354 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:09.171358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:09.171448 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:09.171476 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 20:21:09.171554 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:21:09.171631 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 20:21:09.172226 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.168348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36482 (local address 127.0.116.194:41615)
0926 20:21:09.168618 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:09.168623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:09.168636 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:21:09.168686 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:09.168691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:09.168743 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:09.169119 (+ 376us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:09.169225 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:09.169778 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:09.170744 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:09.171217 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:09.171984 (+ 767us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:09.172047 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:09.172077 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 20:21:09.172131 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":89,"threads_started":1}
I20260926 20:21:09.172814 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.168141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:35253 (local address 127.0.0.1:60006)
0926 20:21:09.168278 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.168291 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:09.168374 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:09.169134 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:09.169137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:09.169152 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:09.169229 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:09.169234 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:09.170172 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:09.170175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:09.171308 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:09.171316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:09.172415 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:09.172419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:09.172515 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:09.172540 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:21:09.172638 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 20:21:09.172707 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 20:21:09.172755 7623 tablet_service.cc:1467] Processing CreateTablet for tablet 4282c0b976214c7f89f35a4e26326dc2 (DEFAULT_TABLE table=test-table [id=b46c079770a74875bfdfe8b2dcfa789b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:21:09.172869 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.168637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60006 (local address 127.0.116.193:35253)
0926 20:21:09.168858 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:09.168863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:09.168876 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:21:09.168928 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:09.168931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:09.168986 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:09.169131 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:09.169319 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:09.170055 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:09.171450 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:09.172226 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:09.172343 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:09.172400 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:09.172668 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 20:21:09.172747 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
I20260926 20:21:09.173087 7623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4282c0b976214c7f89f35a4e26326dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:09.173393 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.168442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:37643 (local address 127.0.0.1:60570)
0926 20:21:09.169244 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.169256 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:09.169336 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:09.169701 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:09.169704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:09.169719 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:09.169800 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:09.169805 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:09.170764 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:09.170767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:09.171878 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:09.172023 (+ 145us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:09.173033 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:09.173036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:09.173125 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:09.173146 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 20:21:09.173214 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 20:21:09.173275 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":86,"threads_started":1}
I20260926 20:21:09.173365 7484 tablet_service.cc:1467] Processing CreateTablet for tablet 4282c0b976214c7f89f35a4e26326dc2 (DEFAULT_TABLE table=test-table [id=b46c079770a74875bfdfe8b2dcfa789b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:21:09.173640 7484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4282c0b976214c7f89f35a4e26326dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:09.173812 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.168877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.0.116.195:37643)
0926 20:21:09.169119 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:09.169123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:09.169412 (+ 289us) server_negotiation.cc:407] Connection header received
0926 20:21:09.169466 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:09.169469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:09.169526 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:09.169601 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:09.169906 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:09.170638 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:09.172141 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:09.172815 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:09.172911 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:09.172970 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:09.173615 (+ 645us) server_negotiation.cc:299] Negotiation successful
0926 20:21:09.173693 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":63,"threads_started":1}
I20260926 20:21:09.174242 7760 tablet_service.cc:1467] Processing CreateTablet for tablet 4282c0b976214c7f89f35a4e26326dc2 (DEFAULT_TABLE table=test-table [id=b46c079770a74875bfdfe8b2dcfa789b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:21:09.174499 7760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4282c0b976214c7f89f35a4e26326dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:09.175964 7923 tablet_bootstrap.cc:492] T 4282c0b976214c7f89f35a4e26326dc2 P 8a913df7cca14f21980b78e45afb13ed: Bootstrap starting.
I20260926 20:21:09.176831 7923 tablet_bootstrap.cc:654] T 4282c0b976214c7f89f35a4e26326dc2 P 8a913df7cca14f21980b78e45afb13ed: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:09.177078 7925 tablet_bootstrap.cc:492] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f: Bootstrap starting.
I20260926 20:21:09.177217 7924 tablet_bootstrap.cc:492] T 4282c0b976214c7f89f35a4e26326dc2 P dbabb7ce4283472aa26a1dc3d52ee4fb: Bootstrap starting.
I20260926 20:21:09.177934 7925 tablet_bootstrap.cc:654] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:09.177987 7924 tablet_bootstrap.cc:654] T 4282c0b976214c7f89f35a4e26326dc2 P dbabb7ce4283472aa26a1dc3d52ee4fb: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:09.178236 7923 tablet_bootstrap.cc:492] T 4282c0b976214c7f89f35a4e26326dc2 P 8a913df7cca14f21980b78e45afb13ed: No bootstrap required, opened a new log
I20260926 20:21:09.178331 7923 ts_tablet_manager.cc:1397] T 4282c0b976214c7f89f35a4e26326dc2 P 8a913df7cca14f21980b78e45afb13ed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:21:09.178834 7923 raft_consensus.cc:348] T 4282c0b976214c7f89f35a4e26326dc2 P 8a913df7cca14f21980b78e45afb13ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a913df7cca14f21980b78e45afb13ed" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41615 } } peers { permanent_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35253 } } peers { permanent_uuid: "65fda696e9f1430abd4889d32512860f" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 37643 } }
I20260926 20:21:09.178953 7923 raft_consensus.cc:374] T 4282c0b976214c7f89f35a4e26326dc2 P 8a913df7cca14f21980b78e45afb13ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:09.178988 7923 raft_consensus.cc:729] T 4282c0b976214c7f89f35a4e26326dc2 P 8a913df7cca14f21980b78e45afb13ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a913df7cca14f21980b78e45afb13ed, State: Initialized, Role: FOLLOWER
I20260926 20:21:09.179164 7923 consensus_queue.cc:260] T 4282c0b976214c7f89f35a4e26326dc2 P 8a913df7cca14f21980b78e45afb13ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a913df7cca14f21980b78e45afb13ed" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41615 } } peers { permanent_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35253 } } peers { permanent_uuid: "65fda696e9f1430abd4889d32512860f" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 37643 } }
I20260926 20:21:09.179235 7924 tablet_bootstrap.cc:492] T 4282c0b976214c7f89f35a4e26326dc2 P dbabb7ce4283472aa26a1dc3d52ee4fb: No bootstrap required, opened a new log
I20260926 20:21:09.179311 7924 ts_tablet_manager.cc:1397] T 4282c0b976214c7f89f35a4e26326dc2 P dbabb7ce4283472aa26a1dc3d52ee4fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:21:09.179545 7923 ts_tablet_manager.cc:1428] T 4282c0b976214c7f89f35a4e26326dc2 P 8a913df7cca14f21980b78e45afb13ed: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 20:21:09.180187 7924 raft_consensus.cc:348] T 4282c0b976214c7f89f35a4e26326dc2 P dbabb7ce4283472aa26a1dc3d52ee4fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a913df7cca14f21980b78e45afb13ed" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41615 } } peers { permanent_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35253 } } peers { permanent_uuid: "65fda696e9f1430abd4889d32512860f" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 37643 } }
I20260926 20:21:09.180307 7924 raft_consensus.cc:374] T 4282c0b976214c7f89f35a4e26326dc2 P dbabb7ce4283472aa26a1dc3d52ee4fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:09.180348 7924 raft_consensus.cc:729] T 4282c0b976214c7f89f35a4e26326dc2 P dbabb7ce4283472aa26a1dc3d52ee4fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbabb7ce4283472aa26a1dc3d52ee4fb, State: Initialized, Role: FOLLOWER
I20260926 20:21:09.180529 7924 consensus_queue.cc:260] T 4282c0b976214c7f89f35a4e26326dc2 P dbabb7ce4283472aa26a1dc3d52ee4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a913df7cca14f21980b78e45afb13ed" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41615 } } peers { permanent_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35253 } } peers { permanent_uuid: "65fda696e9f1430abd4889d32512860f" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 37643 } }
I20260926 20:21:09.180830 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.179956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.116.194:39571)
0926 20:21:09.180562 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.180630 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":108,"threads_started":1}
I20260926 20:21:09.180861 7924 ts_tablet_manager.cc:1428] T 4282c0b976214c7f89f35a4e26326dc2 P dbabb7ce4283472aa26a1dc3d52ee4fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:21:09.182788 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.182519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.116.194:52535)
0926 20:21:09.182662 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.182714 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 20:21:09.182808 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.182365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.116.193:60367)
0926 20:21:09.182630 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.182701 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":106,"threads_started":1}
I20260926 20:21:09.182812 7691 heartbeater.cc:503] Master 127.0.116.253:36501 was elected leader, sending a full tablet report...
I20260926 20:21:09.183673 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.183246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.116.193:42745)
0926 20:21:09.183541 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.183597 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":93,"threads_started":1}
I20260926 20:21:09.184130 7925 tablet_bootstrap.cc:492] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f: No bootstrap required, opened a new log
I20260926 20:21:09.184247 7925 ts_tablet_manager.cc:1397] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260926 20:21:09.184629 7925 raft_consensus.cc:348] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a913df7cca14f21980b78e45afb13ed" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41615 } } peers { permanent_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35253 } } peers { permanent_uuid: "65fda696e9f1430abd4889d32512860f" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 37643 } }
I20260926 20:21:09.184749 7925 raft_consensus.cc:374] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:09.184783 7925 raft_consensus.cc:729] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65fda696e9f1430abd4889d32512860f, State: Initialized, Role: FOLLOWER
I20260926 20:21:09.184912 7925 consensus_queue.cc:260] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a913df7cca14f21980b78e45afb13ed" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41615 } } peers { permanent_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35253 } } peers { permanent_uuid: "65fda696e9f1430abd4889d32512860f" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 37643 } }
I20260926 20:21:09.185173 7925 ts_tablet_manager.cc:1428] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:21:09.185897 7935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.185507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.116.195:53239)
0926 20:21:09.185750 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.185808 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":109,"threads_started":1}
I20260926 20:21:09.186024 7553 heartbeater.cc:503] Master 127.0.116.253:36501 was elected leader, sending a full tablet report...
I20260926 20:21:09.186848 7828 heartbeater.cc:503] Master 127.0.116.253:36501 was elected leader, sending a full tablet report...
I20260926 20:21:09.187191 7935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.186965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.116.195:33869)
0926 20:21:09.187068 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.187127 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 20:21:09.243775 7934 raft_consensus.cc:482] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:21:09.243980 7934 raft_consensus.cc:504] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a913df7cca14f21980b78e45afb13ed" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41615 } } peers { permanent_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35253 } } peers { permanent_uuid: "65fda696e9f1430abd4889d32512860f" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 37643 } }
I20260926 20:21:09.244406 7934 leader_election.cc:288] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a913df7cca14f21980b78e45afb13ed (127.0.116.194:41615), dbabb7ce4283472aa26a1dc3d52ee4fb (127.0.116.193:35253)
I20260926 20:21:09.244928 7504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4282c0b976214c7f89f35a4e26326dc2" candidate_uuid: "65fda696e9f1430abd4889d32512860f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" is_pre_election: true
I20260926 20:21:09.244959 7643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4282c0b976214c7f89f35a4e26326dc2" candidate_uuid: "65fda696e9f1430abd4889d32512860f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a913df7cca14f21980b78e45afb13ed" is_pre_election: true
I20260926 20:21:09.245100 7504 raft_consensus.cc:2454] T 4282c0b976214c7f89f35a4e26326dc2 P dbabb7ce4283472aa26a1dc3d52ee4fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65fda696e9f1430abd4889d32512860f in term 0.
I20260926 20:21:09.245100 7643 raft_consensus.cc:2454] T 4282c0b976214c7f89f35a4e26326dc2 P 8a913df7cca14f21980b78e45afb13ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65fda696e9f1430abd4889d32512860f in term 0.
I20260926 20:21:09.245482 7715 leader_election.cc:302] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65fda696e9f1430abd4889d32512860f, dbabb7ce4283472aa26a1dc3d52ee4fb; no voters:
I20260926 20:21:09.245663 7934 raft_consensus.cc:2781] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:21:09.245720 7934 raft_consensus.cc:482] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:21:09.245787 7934 raft_consensus.cc:3037] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:09.246948 7934 raft_consensus.cc:504] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a913df7cca14f21980b78e45afb13ed" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41615 } } peers { permanent_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35253 } } peers { permanent_uuid: "65fda696e9f1430abd4889d32512860f" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 37643 } }
I20260926 20:21:09.247298 7934 leader_election.cc:288] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f [CANDIDATE]: Term 1 election: Requested vote from peers 8a913df7cca14f21980b78e45afb13ed (127.0.116.194:41615), dbabb7ce4283472aa26a1dc3d52ee4fb (127.0.116.193:35253)
I20260926 20:21:09.247663 7643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4282c0b976214c7f89f35a4e26326dc2" candidate_uuid: "65fda696e9f1430abd4889d32512860f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a913df7cca14f21980b78e45afb13ed"
I20260926 20:21:09.247664 7504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4282c0b976214c7f89f35a4e26326dc2" candidate_uuid: "65fda696e9f1430abd4889d32512860f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb"
I20260926 20:21:09.247798 7643 raft_consensus.cc:3037] T 4282c0b976214c7f89f35a4e26326dc2 P 8a913df7cca14f21980b78e45afb13ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:09.247798 7504 raft_consensus.cc:3037] T 4282c0b976214c7f89f35a4e26326dc2 P dbabb7ce4283472aa26a1dc3d52ee4fb [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:09.249037 7504 raft_consensus.cc:2454] T 4282c0b976214c7f89f35a4e26326dc2 P dbabb7ce4283472aa26a1dc3d52ee4fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65fda696e9f1430abd4889d32512860f in term 1.
I20260926 20:21:09.249037 7643 raft_consensus.cc:2454] T 4282c0b976214c7f89f35a4e26326dc2 P 8a913df7cca14f21980b78e45afb13ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65fda696e9f1430abd4889d32512860f in term 1.
I20260926 20:21:09.249336 7715 leader_election.cc:302] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65fda696e9f1430abd4889d32512860f, dbabb7ce4283472aa26a1dc3d52ee4fb; no voters:
I20260926 20:21:09.249541 7934 raft_consensus.cc:2781] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:21:09.249789 7934 raft_consensus.cc:686] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f [term 1 LEADER]: Becoming Leader. State: Replica: 65fda696e9f1430abd4889d32512860f, State: Running, Role: LEADER
I20260926 20:21:09.249951 7934 consensus_queue.cc:237] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a913df7cca14f21980b78e45afb13ed" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41615 } } peers { permanent_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35253 } } peers { permanent_uuid: "65fda696e9f1430abd4889d32512860f" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 37643 } }
I20260926 20:21:09.250720 7935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.250445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.116.195:33533)
0926 20:21:09.250591 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.250653 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 20:21:09.252238 7188 catalog_manager.cc:5495] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f reported cstate change: term changed from 0 to 1, leader changed from <none> to 65fda696e9f1430abd4889d32512860f (127.0.116.195). New cstate: current_term: 1 leader_uuid: "65fda696e9f1430abd4889d32512860f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a913df7cca14f21980b78e45afb13ed" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65fda696e9f1430abd4889d32512860f" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 37643 } health_report { overall_health: HEALTHY } } }
I20260926 20:21:09.273083 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.269928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.0.116.195:37643)
0926 20:21:09.270058 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:09.270064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:09.270079 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:21:09.270168 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:09.270171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:09.270222 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:09.270306 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:21:09.270556 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:09.271070 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:09.272005 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:09.272237 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:09.272349 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:09.272601 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:21:09.272677 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:21:09.272759 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:09.272880 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 20:21:09.272937 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:21:09.275255 7504 raft_consensus.cc:1261] T 4282c0b976214c7f89f35a4e26326dc2 P dbabb7ce4283472aa26a1dc3d52ee4fb [term 1 FOLLOWER]: Refusing update from remote peer 65fda696e9f1430abd4889d32512860f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:21:09.275324 7643 raft_consensus.cc:1261] T 4282c0b976214c7f89f35a4e26326dc2 P 8a913df7cca14f21980b78e45afb13ed [term 1 FOLLOWER]: Refusing update from remote peer 65fda696e9f1430abd4889d32512860f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:21:09.275868 7936 consensus_queue.cc:1035] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a913df7cca14f21980b78e45afb13ed" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:21:09.276116 7934 consensus_queue.cc:1035] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbabb7ce4283472aa26a1dc3d52ee4fb" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 35253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:21:09.277876 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.277524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.116.193:33207)
0926 20:21:09.277703 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.277766 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260926 20:21:09.278084 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.277761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.116.194:42941)
0926 20:21:09.277954 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.278011 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260926 20:21:09.280493 7939 mvcc.cc:205] Tried to move back new op lower bound from 7333699867746996224 to 7333699867650527232. Current Snapshot: MvccSnapshot[applied={T|T < 7333699867746996224}]
I20260926 20:21:09.281023 7935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.280744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.116.195:36037)
0926 20:21:09.280878 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.280946 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 20:21:09.281060 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.280796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.116.193:37075)
0926 20:21:09.280933 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.280982 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 20:21:09.283779 7940 mvcc.cc:205] Tried to move back new op lower bound from 7333699867746996224 to 7333699867650527232. Current Snapshot: MvccSnapshot[applied={T|T < 7333699867746996224}]
I20260926 20:21:09.284834 7935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.284579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.116.195:37147)
0926 20:21:09.284703 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.284761 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 20:21:09.286396 7188 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:21:09.286554 7188 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:21:09.287397 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.287127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33869 (local address 127.0.116.194:48817)
0926 20:21:09.287237 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.287316 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 20:21:09.290601 7188 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00e0d5ec1e01475a871d550cf2f981e0: Sending DeleteTablet for 3 replicas of tablet 4282c0b976214c7f89f35a4e26326dc2
I20260926 20:21:09.291258 7623 tablet_service.cc:1514] Processing DeleteTablet for tablet 4282c0b976214c7f89f35a4e26326dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:09 UTC) from {username='oryx', principal='oryx/127.0.116.253@KRBTEST.COM'} at 127.0.0.1:36482
I20260926 20:21:09.291540 7622 tablet_service.cc:1514] Processing DeleteTablet for tablet 4282c0b976214c7f89f35a4e26326dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:09 UTC) from {username='oryx', principal='oryx/127.0.116.253@KRBTEST.COM'} at 127.0.0.1:36482
I20260926 20:21:09.291626 7484 tablet_service.cc:1514] Processing DeleteTablet for tablet 4282c0b976214c7f89f35a4e26326dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:09 UTC) from {username='oryx', principal='oryx/127.0.116.253@KRBTEST.COM'} at 127.0.0.1:60006
I20260926 20:21:09.291965 7948 tablet_replica.cc:330] T 4282c0b976214c7f89f35a4e26326dc2 P dbabb7ce4283472aa26a1dc3d52ee4fb: stopping tablet replica
I20260926 20:21:09.292011 7760 tablet_service.cc:1514] Processing DeleteTablet for tablet 4282c0b976214c7f89f35a4e26326dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:09 UTC) from {username='oryx', principal='oryx/127.0.116.253@KRBTEST.COM'} at 127.0.0.1:60570
I20260926 20:21:09.292099 7948 raft_consensus.cc:2229] T 4282c0b976214c7f89f35a4e26326dc2 P dbabb7ce4283472aa26a1dc3d52ee4fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:21:09.292243 7948 raft_consensus.cc:2258] T 4282c0b976214c7f89f35a4e26326dc2 P dbabb7ce4283472aa26a1dc3d52ee4fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:21:09.291906 7947 tablet_replica.cc:330] T 4282c0b976214c7f89f35a4e26326dc2 P 8a913df7cca14f21980b78e45afb13ed: stopping tablet replica
I20260926 20:21:09.292460 7947 raft_consensus.cc:2229] T 4282c0b976214c7f89f35a4e26326dc2 P 8a913df7cca14f21980b78e45afb13ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:21:09.292632 7947 raft_consensus.cc:2258] T 4282c0b976214c7f89f35a4e26326dc2 P 8a913df7cca14f21980b78e45afb13ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:21:09.292600 7949 tablet_replica.cc:330] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f: stopping tablet replica
I20260926 20:21:09.292768 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 7333
I20260926 20:21:09.292773 7949 raft_consensus.cc:2229] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:21:09.293586 7949 raft_consensus.cc:2258] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:21:09.295117 7949 ts_tablet_manager.cc:1905] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:21:09.297691 7949 ts_tablet_manager.cc:1918] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:21:09.297758 7949 log.cc:1196] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/4282c0b976214c7f89f35a4e26326dc2
I20260926 20:21:09.298039 7949 ts_tablet_manager.cc:1939] T 4282c0b976214c7f89f35a4e26326dc2 P 65fda696e9f1430abd4889d32512860f: Deleting consensus metadata
I20260926 20:21:09.298724 7175 catalog_manager.cc:4841] TS 65fda696e9f1430abd4889d32512860f (127.0.116.195:37643): tablet 4282c0b976214c7f89f35a4e26326dc2 (table test-table [id=b46c079770a74875bfdfe8b2dcfa789b]) successfully deleted
I20260926 20:21:09.299768 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 7559
I20260926 20:21:09.299966 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.299676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:35253 (local address 127.0.0.1:60010)
0926 20:21:09.299823 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.299882 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.193:35253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 20:21:09.300136 7947 ts_tablet_manager.cc:1905] T 4282c0b976214c7f89f35a4e26326dc2 P 8a913df7cca14f21980b78e45afb13ed: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 20:21:09.300608 7175 catalog_manager.cc:4584] TS dbabb7ce4283472aa26a1dc3d52ee4fb (127.0.116.193:35253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4282c0b976214c7f89f35a4e26326dc2: Network error: Client connection negotiation failed: client connection to 127.0.116.193:35253: connect: Connection refused (error 111)
I20260926 20:21:09.306871 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 7696
I20260926 20:21:09.306900 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:09.306611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:41615 (local address 127.0.0.1:36496)
0926 20:21:09.306744 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:09.306825 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.194:41615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 20:21:09.313395 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 7049
I20260926 20:21:09.319661 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 7160
2026-09-26T20:21:09Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5594 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:21:09 dist-test-slave-qsm9 krb5kdc[7952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:21:09 dist-test-slave-qsm9 krb5kdc[7952](info): set up 2 sockets
Sep 26 20:21:09 dist-test-slave-qsm9 krb5kdc[7952](info): commencing operation
krb5kdc: starting...
W20260926 20:21:10.372505 467 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:21:10 dist-test-slave-qsm9 krb5kdc[7952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:21:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:21:10Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/oryx_127.0.116.254.keytab.
Entry for principal oryx/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/oryx_127.0.116.254.keytab.
I20260926 20:21:10.499847 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:36149
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/oryx_127.0.116.254.keytab
--principal=oryx/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:36149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:10.614159 7965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:10.614480 7965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:10.614532 7965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:10.618810 7965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:21:10.618897 7965 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 20:21:10.618916 7965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:10.618933 7965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:10.618950 7965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:21:10.618965 7965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:21:10.624595 7965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:36149
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/oryx_127.0.116.254.keytab
--principal=oryx/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:36149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:10.625811 7965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:10.626911 7965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:21:10.636029 7965 server_base.cc:1034] running on GCE node
W20260926 20:21:10.635977 7971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:10.636018 7970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:10.635947 7973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:10.636775 7965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:10.638036 7965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:10.639266 7965 hybrid_clock.cc:648] HybridClock initialized: now 1790454070639231 us; error 49 us; skew 500 ppm
Sep 26 20:21:10 dist-test-slave-qsm9 krb5kdc[7952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454070, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:10.642395 7965 init.cc:377] Logged in from keytab as oryx/127.0.116.254@KRBTEST.COM (short username oryx)
I20260926 20:21:10.643476 7965 webserver.cc:460] Webserver started at http://127.0.116.254:33141/ using document root <none> and password file <none>
I20260926 20:21:10.644125 7965 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:10.644220 7965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:10.644454 7965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:10.646579 7965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "d5832e6267154118849270dd5bfccfc9"
format_stamp: "Formatted at 2026-09-26 20:21:10 on dist-test-slave-qsm9"
server_key: "93f8eab51bf0dce3ebf4d8790af6c07b"
server_key_iv: "f9e1392fac7770348d0870c307a5a679"
server_key_version: "encryptionkey@0"
I20260926 20:21:10.647193 7965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "d5832e6267154118849270dd5bfccfc9"
format_stamp: "Formatted at 2026-09-26 20:21:10 on dist-test-slave-qsm9"
server_key: "93f8eab51bf0dce3ebf4d8790af6c07b"
server_key_iv: "f9e1392fac7770348d0870c307a5a679"
server_key_version: "encryptionkey@0"
I20260926 20:21:10.651273 7965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260926 20:21:10.653955 7980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:10.655471 7965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:21:10.655730 7965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "d5832e6267154118849270dd5bfccfc9"
format_stamp: "Formatted at 2026-09-26 20:21:10 on dist-test-slave-qsm9"
server_key: "93f8eab51bf0dce3ebf4d8790af6c07b"
server_key_iv: "f9e1392fac7770348d0870c307a5a679"
server_key_version: "encryptionkey@0"
I20260926 20:21:10.655866 7965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:10.673439 7965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:10.676864 7965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:10.677093 7965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:10.685112 7965 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:36149
I20260926 20:21:10.685130 8042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:36149 every 8 connection(s)
I20260926 20:21:10.686122 7965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:21:10.686782 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 7965
I20260926 20:21:10.686928 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:21:10.687243 467 external_mini_cluster.cc:1428] Setting key b9d2c09f31daf6c9c1def25320dcea51
I20260926 20:21:10.690317 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 20:21:10 dist-test-slave-qsm9 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM
I20260926 20:21:10.697368 8043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9: Bootstrap starting.
I20260926 20:21:10.699599 8043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:10.700654 8043 log.cc:824] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9: Log is configured to *not* fsync() on all Append() calls
I20260926 20:21:10.702188 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:10.688792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49834 (local address 127.0.116.254:36149)
0926 20:21:10.689208 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:10.689224 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:10.689262 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:21:10.689939 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:10.689956 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:10.690365 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:10.690705 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:10.691619 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:10.692708 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:10.693897 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:10.694384 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:10.694502 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:10.697393 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:10.697397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:10.697410 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:10.697461 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:10.699223 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:10.699823 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:10.699828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:10.699833 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:10.699943 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:10.700241 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:10.700249 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:10.700250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:10.700646 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:10.700848 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:10.701160 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 20:21:10.701452 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":111,"threads_started":1}
I20260926 20:21:10.702742 8043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9: No bootstrap required, opened a new log
I20260926 20:21:10.706205 8043 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5832e6267154118849270dd5bfccfc9" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36149 } }
I20260926 20:21:10.706563 8043 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:10.706647 8043 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5832e6267154118849270dd5bfccfc9, State: Initialized, Role: FOLLOWER
I20260926 20:21:10.707108 8043 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5832e6267154118849270dd5bfccfc9" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36149 } }
I20260926 20:21:10.707230 8043 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:21:10.707320 8043 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:21:10.707424 8043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:10.708453 8043 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5832e6267154118849270dd5bfccfc9" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36149 } }
I20260926 20:21:10.708868 8043 leader_election.cc:302] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5832e6267154118849270dd5bfccfc9; no voters:
I20260926 20:21:10.709292 8043 leader_election.cc:288] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:21:10.709422 8048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:21:10.709674 8048 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9 [term 1 LEADER]: Becoming Leader. State: Replica: d5832e6267154118849270dd5bfccfc9, State: Running, Role: LEADER
I20260926 20:21:10.710040 8048 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5832e6267154118849270dd5bfccfc9" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36149 } }
I20260926 20:21:10.710695 8043 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:21:10.711840 8050 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5832e6267154118849270dd5bfccfc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5832e6267154118849270dd5bfccfc9" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36149 } } }
I20260926 20:21:10.711970 8050 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:10.712250 8049 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5832e6267154118849270dd5bfccfc9. Latest consensus state: current_term: 1 leader_uuid: "d5832e6267154118849270dd5bfccfc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5832e6267154118849270dd5bfccfc9" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36149 } } }
I20260926 20:21:10.712342 8049 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:10.712445 8057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:21:10.715966 8057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:21:10.721913 8057 catalog_manager.cc:1347] Generated new cluster ID: e56c4ff373a94d3f89807086207a4666
I20260926 20:21:10.722007 8057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:21:10.737057 8057 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:21:10.738430 8057 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:21:10.745855 8057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9: Generated new TSK 0
I20260926 20:21:10.746740 8057 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/oryx_127.0.116.193.keytab.
Entry for principal oryx/127.0.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/oryx_127.0.116.193.keytab.
I20260926 20:21:10.777107 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:36149
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/oryx_127.0.116.193.keytab
--principal=oryx/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:10.892844 8069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:10.893137 8069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:10.893224 8069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:10.897547 8069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:10.897639 8069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:10.897725 8069 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:21:10.897818 8069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:21:10.903575 8069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/oryx_127.0.116.193.keytab
--principal=oryx/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.116.254:36149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:10.904808 8069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:10.905958 8069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:10.916334 8077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:10.916505 8069 server_base.cc:1034] running on GCE node
W20260926 20:21:10.916589 8074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:10.916680 8075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:10.917172 8069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:10.917871 8069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:10.919090 8069 hybrid_clock.cc:648] HybridClock initialized: now 1790454070919056 us; error 48 us; skew 500 ppm
Sep 26 20:21:10 dist-test-slave-qsm9 krb5kdc[7952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454070, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:10.922092 8069 init.cc:377] Logged in from keytab as oryx/127.0.116.193@KRBTEST.COM (short username oryx)
I20260926 20:21:10.923134 8069 webserver.cc:460] Webserver started at http://127.0.116.193:44971/ using document root <none> and password file <none>
I20260926 20:21:10.923834 8069 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:10.923914 8069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:10.924167 8069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:10.926385 8069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "748991b49d7948afa8b7227a4886ad38"
format_stamp: "Formatted at 2026-09-26 20:21:10 on dist-test-slave-qsm9"
server_key: "13bf9c4f3bca91df63b8d8b16b2fbf90"
server_key_iv: "5edf46ac84fcefa7e558220d01bd9459"
server_key_version: "encryptionkey@0"
I20260926 20:21:10.927060 8069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "748991b49d7948afa8b7227a4886ad38"
format_stamp: "Formatted at 2026-09-26 20:21:10 on dist-test-slave-qsm9"
server_key: "13bf9c4f3bca91df63b8d8b16b2fbf90"
server_key_iv: "5edf46ac84fcefa7e558220d01bd9459"
server_key_version: "encryptionkey@0"
I20260926 20:21:10.931644 8069 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 20:21:10.934461 8084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:10.935927 8069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:21:10.936087 8069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "748991b49d7948afa8b7227a4886ad38"
format_stamp: "Formatted at 2026-09-26 20:21:10 on dist-test-slave-qsm9"
server_key: "13bf9c4f3bca91df63b8d8b16b2fbf90"
server_key_iv: "5edf46ac84fcefa7e558220d01bd9459"
server_key_version: "encryptionkey@0"
I20260926 20:21:10.936224 8069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:10.961992 8069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:10.965646 8069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:10.965862 8069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:10.967475 8069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:10.967610 8069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:10.967685 8069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:10.967733 8069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:21:10 dist-test-slave-qsm9 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454070, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM
I20260926 20:21:10.978866 8069 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:38473
I20260926 20:21:10.978984 8199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:38473 every 8 connection(s)
I20260926 20:21:10.980141 8069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:21:10.982214 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:10.968311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:37585 (local address 127.0.116.254:36149)
0926 20:21:10.968480 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:10.968485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:10.969350 (+ 865us) server_negotiation.cc:407] Connection header received
0926 20:21:10.970325 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:10.970329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:10.970388 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:10.970471 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:10.972023 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:10.972568 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:10.973806 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:10.974086 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:10.974210 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:10.978237 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:10.978242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:10.978245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:10.978275 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:10.980078 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:10.980739 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:10.980743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:10.980745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:10.980845 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:10.981192 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:10.981196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:10.981198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:10.981418 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:10.981502 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:10.981962 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 20:21:10.982078 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:21:10.983024 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:10.968714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:36149 (local address 127.0.116.193:37585)
0926 20:21:10.969175 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:10.969226 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:10.970109 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:10.970808 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:10.970817 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:10.971221 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:10.971872 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:10.971885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:10.972703 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:10.972711 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:10.973672 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:10.973682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:10.974260 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:10.974264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:10.974463 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:10.975232 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:10.975258 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:10.978033 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:10.980244 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:10.980252 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:10.980269 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:10.980606 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:10.980953 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:10.980956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:10.980958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:10.981092 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:10.981512 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:10.981517 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:10.981760 (+ 243us) client_negotiation.cc:769] Sending connection context
0926 20:21:10.981912 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 20:21:10.982179 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":141,"threads_started":1}
I20260926 20:21:10.984485 8200 heartbeater.cc:347] Connected to a master server at 127.0.116.254:36149
I20260926 20:21:10.984671 8200 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:10.984686 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 8069
I20260926 20:21:10.984803 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:21:10.985167 467 external_mini_cluster.cc:1428] Setting key 3995b66511e0bbf54992f29b410595ba
I20260926 20:21:10.985294 8200 heartbeater.cc:511] Master 127.0.116.254:36149 requested a full tablet report, sending...
I20260926 20:21:10.987222 7997 ts_manager.cc:194] Registered new tserver with Master: 748991b49d7948afa8b7227a4886ad38 (127.0.116.193:38473)
I20260926 20:21:10.988932 7997 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:37585
I20260926 20:21:10.996055 8206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:10.986891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:36149 (local address 127.0.116.193:43041)
0926 20:21:10.987668 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:10.987680 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:10.987793 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:10.988048 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:10.988050 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:10.988265 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:10.988362 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:10.988368 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:10.989369 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:10.989374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:10.990356 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:10.990367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:10.990976 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:10.990979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:10.991068 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:10.991617 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:10.991630 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:10.991980 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:10.994111 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:10.994114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:10.994116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:10.994345 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:10.994689 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:10.994693 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:10.994695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:10.994819 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:10.995339 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:10.995345 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:10.995465 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 20:21:10.995647 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 20:21:10.995819 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":1,"thread_start_us":124,"threads_started":1}
I20260926 20:21:10.996599 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:10.986975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:43041 (local address 127.0.116.254:36149)
0926 20:21:10.987110 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:10.987117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:10.987749 (+ 632us) server_negotiation.cc:407] Connection header received
0926 20:21:10.987894 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:10.987897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:10.987940 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:10.988039 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:10.988472 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:10.989206 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:10.990474 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:10.990708 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:10.990826 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:10.992141 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:10.992150 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:10.992155 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:10.992189 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:10.993978 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:10.994457 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:10.994463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:10.994466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:10.994576 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:10.994928 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:10.994931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:10.994933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:10.995221 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:10.995331 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:10.995801 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 20:21:10.996039 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":54}
WARNING: no policy specified for oryx/127.0.116.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.116.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/oryx_127.0.116.194.keytab.
Entry for principal oryx/127.0.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/oryx_127.0.116.194.keytab.
I20260926 20:21:11.016762 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:36149
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/oryx_127.0.116.194.keytab
--principal=oryx/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:11.131454 8209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:11.131785 8209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:11.131877 8209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:11.136034 8209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:11.136107 8209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:11.136152 8209 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:21:11.136188 8209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:21:11.141758 8209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/oryx_127.0.116.194.keytab
--principal=oryx/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.116.254:36149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:11.142974 8209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:11.144100 8209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:11.154778 8215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:11.154760 8214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:11.154767 8217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:11.154915 8209 server_base.cc:1034] running on GCE node
I20260926 20:21:11.155788 8209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:11.156589 8209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:11.157807 8209 hybrid_clock.cc:648] HybridClock initialized: now 1790454071157793 us; error 29 us; skew 500 ppm
Sep 26 20:21:11 dist-test-slave-qsm9 krb5kdc[7952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454071, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:11.160904 8209 init.cc:377] Logged in from keytab as oryx/127.0.116.194@KRBTEST.COM (short username oryx)
I20260926 20:21:11.161820 8209 webserver.cc:460] Webserver started at http://127.0.116.194:36413/ using document root <none> and password file <none>
I20260926 20:21:11.162444 8209 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:11.162513 8209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:11.162688 8209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:11.164848 8209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "2caca33540084b7ba7288fbe0b70af32"
format_stamp: "Formatted at 2026-09-26 20:21:11 on dist-test-slave-qsm9"
server_key: "9bd525e676cc6572cef313ec453bb93d"
server_key_iv: "53dcf1455f3c3af67ced37d751c96012"
server_key_version: "encryptionkey@0"
I20260926 20:21:11.165400 8209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "2caca33540084b7ba7288fbe0b70af32"
format_stamp: "Formatted at 2026-09-26 20:21:11 on dist-test-slave-qsm9"
server_key: "9bd525e676cc6572cef313ec453bb93d"
server_key_iv: "53dcf1455f3c3af67ced37d751c96012"
server_key_version: "encryptionkey@0"
I20260926 20:21:11.169485 8209 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:21:11.172132 8224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:11.173467 8209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:21:11.173573 8209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "2caca33540084b7ba7288fbe0b70af32"
format_stamp: "Formatted at 2026-09-26 20:21:11 on dist-test-slave-qsm9"
server_key: "9bd525e676cc6572cef313ec453bb93d"
server_key_iv: "53dcf1455f3c3af67ced37d751c96012"
server_key_version: "encryptionkey@0"
I20260926 20:21:11.173671 8209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:11.194587 8209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:11.198225 8209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:11.198396 8209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:11.199950 8209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:11.200042 8209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:11.200124 8209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:11.200167 8209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:21:11 dist-test-slave-qsm9 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454071, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM
I20260926 20:21:11.211895 8209 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:42217
I20260926 20:21:11.212260 8339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:42217 every 8 connection(s)
I20260926 20:21:11.213037 8209 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:21:11.213369 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 8209
I20260926 20:21:11.213505 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:21:11.213852 467 external_mini_cluster.cc:1428] Setting key b1ff0fcc5ce64f58e4d939c66f119317
I20260926 20:21:11.215400 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:11.200771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:33333 (local address 127.0.116.254:36149)
0926 20:21:11.200947 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:11.200953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:11.202043 (+ 1090us) server_negotiation.cc:407] Connection header received
0926 20:21:11.203043 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:11.203047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:11.203100 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:11.203159 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:11.204861 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.205439 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.206662 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.206884 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.206999 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.210873 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:11.210877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:11.210881 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:11.210917 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:11.212957 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:11.213578 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:11.213582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:11.213583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:11.213682 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:11.214127 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:11.214130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:11.214131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:11.214401 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:11.214498 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:11.215125 (+ 627us) server_negotiation.cc:299] Negotiation successful
0926 20:21:11.215276 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:21:11.216629 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:11.201257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:36149 (local address 127.0.116.194:33333)
0926 20:21:11.201867 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:11.201922 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:11.202783 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:11.203664 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:11.203674 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:11.204130 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:11.204691 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:11.204712 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.205578 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.205582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:11.206497 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:11.206506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.207044 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.207048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:11.207274 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.208101 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:11.208128 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:11.210638 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:11.213096 (+ 2458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:11.213107 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:11.213123 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:11.213429 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:11.213839 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:11.213842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:11.213844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:11.214015 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:11.214495 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:11.214502 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:11.214826 (+ 324us) client_negotiation.cc:769] Sending connection context
0926 20:21:11.215058 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 20:21:11.215452 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":135,"threads_started":1}
I20260926 20:21:11.218094 8340 heartbeater.cc:347] Connected to a master server at 127.0.116.254:36149
I20260926 20:21:11.218240 8340 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:11.218796 8340 heartbeater.cc:511] Master 127.0.116.254:36149 requested a full tablet report, sending...
I20260926 20:21:11.220041 7995 ts_manager.cc:194] Registered new tserver with Master: 2caca33540084b7ba7288fbe0b70af32 (127.0.116.194:42217)
I20260926 20:21:11.220964 7995 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:33333
WARNING: no policy specified for oryx/127.0.116.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.116.195@KRBTEST.COM" created.
I20260926 20:21:11.229156 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:11.220580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:33407 (local address 127.0.116.254:36149)
0926 20:21:11.220687 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:11.220691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:11.221014 (+ 323us) server_negotiation.cc:407] Connection header received
0926 20:21:11.221162 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:11.221164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:11.221214 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:11.221297 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:11.221827 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.222730 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.223823 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.224200 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.224333 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.225601 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:11.225605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:11.225607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:11.225633 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:11.227127 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:11.227727 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:11.227730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:11.227732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:11.227814 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:11.228118 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:11.228123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:11.228124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:11.228417 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:11.228536 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:11.228836 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 20:21:11.228958 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 20:21:11.229153 8347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:11.220540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:36149 (local address 127.0.116.194:33407)
0926 20:21:11.220928 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:11.220940 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:11.221070 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:11.221312 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:11.221315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:11.221536 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:11.221691 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:11.221698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.222887 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.222890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:11.223706 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:11.223715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.224333 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.224335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:11.224414 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.225013 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:11.225024 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:11.225414 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:11.227242 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:11.227245 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:11.227247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:11.227584 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:11.227924 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:11.227927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:11.227929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:11.228025 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:11.228529 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:11.228533 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:11.228643 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 20:21:11.228767 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 20:21:11.228945 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":220,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/oryx_127.0.116.195.keytab.
Entry for principal oryx/127.0.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/oryx_127.0.116.195.keytab.
I20260926 20:21:11.247306 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:36149
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/oryx_127.0.116.195.keytab
--principal=oryx/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:11.364331 8349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:11.364674 8349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:11.364763 8349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:11.369057 8349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:11.369170 8349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:11.369246 8349 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:21:11.369334 8349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:21:11.374979 8349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/krb5kdc/oryx_127.0.116.195.keytab
--principal=oryx/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.116.254:36149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:11.376209 8349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:11.377404 8349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:11.388249 8354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:11.388306 8355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:11.388306 8357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:11.388440 8349 server_base.cc:1034] running on GCE node
I20260926 20:21:11.389093 8349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:11.389819 8349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:11.391028 8349 hybrid_clock.cc:648] HybridClock initialized: now 1790454071390992 us; error 50 us; skew 500 ppm
Sep 26 20:21:11 dist-test-slave-qsm9 krb5kdc[7952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454071, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:11.394609 8349 init.cc:377] Logged in from keytab as oryx/127.0.116.195@KRBTEST.COM (short username oryx)
I20260926 20:21:11.396114 8349 webserver.cc:460] Webserver started at http://127.0.116.195:39007/ using document root <none> and password file <none>
I20260926 20:21:11.396945 8349 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:11.397045 8349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:11.397302 8349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:11.400264 8349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "af9fef83ce7b4f1faf748eff13d7fd3a"
format_stamp: "Formatted at 2026-09-26 20:21:11 on dist-test-slave-qsm9"
server_key: "729ddb516e20b66737a0f49f8ce1c1d9"
server_key_iv: "20ed7be88bd6459d51c8007cc72cf577"
server_key_version: "encryptionkey@0"
I20260926 20:21:11.400805 8349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "af9fef83ce7b4f1faf748eff13d7fd3a"
format_stamp: "Formatted at 2026-09-26 20:21:11 on dist-test-slave-qsm9"
server_key: "729ddb516e20b66737a0f49f8ce1c1d9"
server_key_iv: "20ed7be88bd6459d51c8007cc72cf577"
server_key_version: "encryptionkey@0"
I20260926 20:21:11.404666 8349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:21:11.407244 8364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:11.408644 8349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:21:11.408782 8349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "af9fef83ce7b4f1faf748eff13d7fd3a"
format_stamp: "Formatted at 2026-09-26 20:21:11 on dist-test-slave-qsm9"
server_key: "729ddb516e20b66737a0f49f8ce1c1d9"
server_key_iv: "20ed7be88bd6459d51c8007cc72cf577"
server_key_version: "encryptionkey@0"
I20260926 20:21:11.408914 8349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:11.428369 8349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:11.432098 8349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:11.432313 8349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:11.434237 8349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:11.434310 8349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:11.434381 8349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:11.434412 8349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:21:11 dist-test-slave-qsm9 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454071, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM
I20260926 20:21:11.446115 8349 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:46823
I20260926 20:21:11.446312 8479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:46823 every 8 connection(s)
I20260926 20:21:11.447409 8349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:21:11.449561 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:11.435063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:55205 (local address 127.0.116.254:36149)
0926 20:21:11.435215 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:11.435219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:11.436389 (+ 1170us) server_negotiation.cc:407] Connection header received
0926 20:21:11.437583 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:11.437588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:11.437664 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:11.437770 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:11.439465 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.440057 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.441307 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.441626 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.441777 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.445360 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:11.445363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:11.445366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:11.445396 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:11.447242 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:11.447906 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:11.447909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:11.447911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:11.448020 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:11.448449 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:11.448453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:11.448455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:11.448734 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:11.448822 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:11.449307 (+ 485us) server_negotiation.cc:299] Negotiation successful
0926 20:21:11.449418 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:21:11.450400 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:11.435688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:36149 (local address 127.0.116.195:55205)
0926 20:21:11.436222 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:11.436267 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:11.437277 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:11.438032 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:11.438044 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:11.438544 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:11.439291 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:11.439316 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.440204 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.440209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:11.441167 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:11.441176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.441848 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.441856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:11.442123 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.442892 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:11.442913 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:11.445134 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:11.447387 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:11.447392 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:11.447405 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:11.447781 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:11.448134 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:11.448137 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:11.448139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:11.448317 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:11.448819 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:11.448824 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:11.449069 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 20:21:11.449219 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 20:21:11.449479 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":143,"threads_started":1}
I20260926 20:21:11.451733 8480 heartbeater.cc:347] Connected to a master server at 127.0.116.254:36149
I20260926 20:21:11.451879 8480 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:11.452507 8480 heartbeater.cc:511] Master 127.0.116.254:36149 requested a full tablet report, sending...
I20260926 20:21:11.453676 7995 ts_manager.cc:194] Registered new tserver with Master: af9fef83ce7b4f1faf748eff13d7fd3a (127.0.116.195:46823)
I20260926 20:21:11.454447 7995 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:55205
I20260926 20:21:11.454638 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 8349
I20260926 20:21:11.454771 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:21:11.455152 467 external_mini_cluster.cc:1428] Setting key 58b7f17b440a9c4d1d8adeb5a6cbebf3
I20260926 20:21:11.457399 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:21:11.462132 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:11.454219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:55021 (local address 127.0.116.254:36149)
0926 20:21:11.454347 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:11.454351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:11.454581 (+ 230us) server_negotiation.cc:407] Connection header received
0926 20:21:11.454777 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:11.454783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:11.454834 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:11.454886 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:11.455578 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.456092 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.457236 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.457557 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.457656 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.458776 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:11.458782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:11.458788 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:11.458822 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:11.460506 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:11.460935 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:11.460938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:11.460940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:11.461025 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:11.461306 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:11.461309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:11.461311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:11.461493 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:11.461595 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:11.461859 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 20:21:11.461978 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:21:11.462144 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:11.454083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:36149 (local address 127.0.116.195:55021)
0926 20:21:11.454488 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:11.454504 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:11.454657 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:11.455036 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:11.455038 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:11.455263 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:11.455441 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:11.455464 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.456233 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.456237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:11.457114 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:11.457124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.457669 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.457672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:11.457745 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.458286 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:11.458303 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:11.458635 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:11.460619 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:11.460624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:11.460626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:11.460829 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:11.461119 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:11.461122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:11.461124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:11.461210 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:11.461593 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:11.461596 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:11.461693 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 20:21:11.461804 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 20:21:11.461965 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":1,"thread_start_us":112,"threads_started":1}
Sep 26 20:21:11 dist-test-slave-qsm9 krb5kdc[7952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454071, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:21:11 dist-test-slave-qsm9 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:11 dist-test-slave-qsm9 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:21:11.482990 8046 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:11.476491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52480 (local address 127.0.116.254:36149)
0926 20:21:11.476637 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:11.476642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:11.476663 (+ 21us) server_negotiation.cc:407] Connection header received
0926 20:21:11.476707 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:11.476709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:11.476750 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:11.476824 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:11.477722 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.478348 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.479184 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.479389 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.479465 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.482801 (+ 3336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52480: BlockingRecv error: recv got EOF from 127.0.0.1:52480 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 20:21:11 dist-test-slave-qsm9 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454071, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM
I20260926 20:21:11.493191 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:11.484588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52484 (local address 127.0.116.254:36149)
0926 20:21:11.484709 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:11.484714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:11.484853 (+ 139us) server_negotiation.cc:407] Connection header received
0926 20:21:11.484939 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:11.484941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:11.484980 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:11.485049 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:11.485704 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.486230 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.487099 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.487301 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.487375 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.489818 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:11.489822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:11.489824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:11.489851 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:11.491238 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:11.491730 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:11.491735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:11.491737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:11.491842 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:11.492228 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:11.492234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:11.492236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:11.492453 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:11.492610 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:11.492874 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 20:21:11.492997 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:21:11.496450 7995 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:21:11.499543 7995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:21:11.514835 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:11.508474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.0.116.193:38473)
0926 20:21:11.509604 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:11.509612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:11.509636 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:21:11.509716 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:11.509721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:11.509934 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:11.510090 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:11.510429 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.511769 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.513463 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.514120 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.514246 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.514403 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:11.514584 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 20:21:11.514685 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1016,"thread_start_us":75,"threads_started":1}
I20260926 20:21:11.514837 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:11.508359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:38473 (local address 127.0.0.1:38166)
0926 20:21:11.509174 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:11.509228 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:11.509363 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:11.510097 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:11.510103 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:11.510127 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:11.510270 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:11.510277 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.511902 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.511908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:11.513346 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:11.513353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.514250 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.514254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:11.514368 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.514404 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 20:21:11.514551 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 20:21:11.514685 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":129,"spinlock_wait_cycles":7936,"thread_start_us":132,"threads_started":1}
I20260926 20:21:11.515065 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:11.508632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:42217 (local address 127.0.0.1:45314)
0926 20:21:11.509580 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:11.509594 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:11.509691 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:11.510382 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:11.510385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:11.510398 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:11.510487 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:11.510493 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.511970 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.511973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:11.513372 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:11.513385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.514627 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.514630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:11.514726 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.514756 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 20:21:11.514817 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 20:21:11.514889 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":100,"thread_start_us":84,"threads_started":1}
I20260926 20:21:11.515560 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:11.509150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45314 (local address 127.0.116.194:42217)
0926 20:21:11.509924 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:11.509933 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:11.509951 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:21:11.510010 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:11.510039 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:11.510243 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:11.510400 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:11.510663 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.511856 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.513518 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.514350 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.514439 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.514578 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:11.515294 (+ 716us) server_negotiation.cc:299] Negotiation successful
0926 20:21:11.515396 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":116,"threads_started":1}
I20260926 20:21:11.515853 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:11.509280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59368 (local address 127.0.116.195:46823)
0926 20:21:11.509649 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:11.509660 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:11.509678 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:21:11.509740 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:11.509744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:11.509944 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:11.510089 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:11.510524 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.511291 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.513287 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.514147 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.514309 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.514481 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:11.515399 (+ 918us) server_negotiation.cc:299] Negotiation successful
0926 20:21:11.515521 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":92,"threads_started":1}
I20260926 20:21:11.516474 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:11.508375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:46823 (local address 127.0.0.1:59368)
0926 20:21:11.509316 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:11.509328 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:11.509395 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:11.510253 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:11.510255 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:11.510269 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:11.510381 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:11.510387 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.511408 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.511414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:11.513046 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:11.513059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.514547 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.514551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:11.514689 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.514713 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:21:11.514784 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 20:21:11.516347 (+ 1563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":1493,"thread_start_us":92,"threads_started":1}
I20260926 20:21:11.518224 8274 tablet_service.cc:1467] Processing CreateTablet for tablet a69424bf028a45ddb2e48b000259cf14 (DEFAULT_TABLE table=test-table [id=316f48f302e146ebb3aeac59768df4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:21:11.518234 8134 tablet_service.cc:1467] Processing CreateTablet for tablet a69424bf028a45ddb2e48b000259cf14 (DEFAULT_TABLE table=test-table [id=316f48f302e146ebb3aeac59768df4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:21:11.519304 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a69424bf028a45ddb2e48b000259cf14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:11.519304 8134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a69424bf028a45ddb2e48b000259cf14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:11.519081 8414 tablet_service.cc:1467] Processing CreateTablet for tablet a69424bf028a45ddb2e48b000259cf14 (DEFAULT_TABLE table=test-table [id=316f48f302e146ebb3aeac59768df4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:21:11.520184 8414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a69424bf028a45ddb2e48b000259cf14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:11.526044 8508 tablet_bootstrap.cc:492] T a69424bf028a45ddb2e48b000259cf14 P 2caca33540084b7ba7288fbe0b70af32: Bootstrap starting.
I20260926 20:21:11.527462 8509 tablet_bootstrap.cc:492] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38: Bootstrap starting.
I20260926 20:21:11.528620 8508 tablet_bootstrap.cc:654] T a69424bf028a45ddb2e48b000259cf14 P 2caca33540084b7ba7288fbe0b70af32: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:11.529520 8508 log.cc:824] T a69424bf028a45ddb2e48b000259cf14 P 2caca33540084b7ba7288fbe0b70af32: Log is configured to *not* fsync() on all Append() calls
I20260926 20:21:11.529644 8509 tablet_bootstrap.cc:654] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:11.530208 8510 tablet_bootstrap.cc:492] T a69424bf028a45ddb2e48b000259cf14 P af9fef83ce7b4f1faf748eff13d7fd3a: Bootstrap starting.
I20260926 20:21:11.530568 8509 log.cc:824] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38: Log is configured to *not* fsync() on all Append() calls
I20260926 20:21:11.531809 8508 tablet_bootstrap.cc:492] T a69424bf028a45ddb2e48b000259cf14 P 2caca33540084b7ba7288fbe0b70af32: No bootstrap required, opened a new log
I20260926 20:21:11.532023 8508 ts_tablet_manager.cc:1397] T a69424bf028a45ddb2e48b000259cf14 P 2caca33540084b7ba7288fbe0b70af32: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:21:11.532377 8510 tablet_bootstrap.cc:654] T a69424bf028a45ddb2e48b000259cf14 P af9fef83ce7b4f1faf748eff13d7fd3a: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:11.532796 8509 tablet_bootstrap.cc:492] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38: No bootstrap required, opened a new log
I20260926 20:21:11.533005 8509 ts_tablet_manager.cc:1397] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 20:21:11.533176 8510 log.cc:824] T a69424bf028a45ddb2e48b000259cf14 P af9fef83ce7b4f1faf748eff13d7fd3a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:21:11.535293 8510 tablet_bootstrap.cc:492] T a69424bf028a45ddb2e48b000259cf14 P af9fef83ce7b4f1faf748eff13d7fd3a: No bootstrap required, opened a new log
I20260926 20:21:11.535540 8510 ts_tablet_manager.cc:1397] T a69424bf028a45ddb2e48b000259cf14 P af9fef83ce7b4f1faf748eff13d7fd3a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 20:21:11.536346 8508 raft_consensus.cc:348] T a69424bf028a45ddb2e48b000259cf14 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } }
I20260926 20:21:11.536662 8508 raft_consensus.cc:374] T a69424bf028a45ddb2e48b000259cf14 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:11.536720 8508 raft_consensus.cc:729] T a69424bf028a45ddb2e48b000259cf14 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2caca33540084b7ba7288fbe0b70af32, State: Initialized, Role: FOLLOWER
I20260926 20:21:11.537189 8508 consensus_queue.cc:260] T a69424bf028a45ddb2e48b000259cf14 P 2caca33540084b7ba7288fbe0b70af32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } }
I20260926 20:21:11.537537 8509 raft_consensus.cc:348] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } }
I20260926 20:21:11.537791 8509 raft_consensus.cc:374] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:11.538029 8340 heartbeater.cc:503] Master 127.0.116.254:36149 was elected leader, sending a full tablet report...
I20260926 20:21:11.538254 8508 ts_tablet_manager.cc:1428] T a69424bf028a45ddb2e48b000259cf14 P 2caca33540084b7ba7288fbe0b70af32: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:21:11.537868 8509 raft_consensus.cc:729] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 748991b49d7948afa8b7227a4886ad38, State: Initialized, Role: FOLLOWER
I20260926 20:21:11.539039 8510 raft_consensus.cc:348] T a69424bf028a45ddb2e48b000259cf14 P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } }
I20260926 20:21:11.539290 8510 raft_consensus.cc:374] T a69424bf028a45ddb2e48b000259cf14 P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:11.539350 8510 raft_consensus.cc:729] T a69424bf028a45ddb2e48b000259cf14 P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9fef83ce7b4f1faf748eff13d7fd3a, State: Initialized, Role: FOLLOWER
I20260926 20:21:11.539418 8509 consensus_queue.cc:260] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } }
I20260926 20:21:11.539836 8510 consensus_queue.cc:260] T a69424bf028a45ddb2e48b000259cf14 P af9fef83ce7b4f1faf748eff13d7fd3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } }
I20260926 20:21:11.540062 8200 heartbeater.cc:503] Master 127.0.116.254:36149 was elected leader, sending a full tablet report...
I20260926 20:21:11.540402 8509 ts_tablet_manager.cc:1428] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 20:21:11.540899 8510 ts_tablet_manager.cc:1428] T a69424bf028a45ddb2e48b000259cf14 P af9fef83ce7b4f1faf748eff13d7fd3a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 20:21:11.541245 8480 heartbeater.cc:503] Master 127.0.116.254:36149 was elected leader, sending a full tablet report...
I20260926 20:21:11.587539 8515 raft_consensus.cc:482] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:21:11.587769 8515 raft_consensus.cc:504] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } }
I20260926 20:21:11.588922 8515 leader_election.cc:288] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af9fef83ce7b4f1faf748eff13d7fd3a (127.0.116.195:46823), 2caca33540084b7ba7288fbe0b70af32 (127.0.116.194:42217)
I20260926 20:21:11.593519 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:11.589438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:49819 (local address 127.0.116.194:42217)
0926 20:21:11.589628 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:11.589634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:11.589815 (+ 181us) server_negotiation.cc:407] Connection header received
0926 20:21:11.589963 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:11.589966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:11.590014 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:11.590084 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:11.590454 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.591018 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.592156 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.592795 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.592935 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.592988 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:11.593315 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 20:21:11.593389 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260926 20:21:11.593519 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:11.589350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:59131 (local address 127.0.116.195:46823)
0926 20:21:11.589512 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:11.589517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:11.589767 (+ 250us) server_negotiation.cc:407] Connection header received
0926 20:21:11.589895 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:11.589899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:11.589960 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:11.590060 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:11.590334 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.590993 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.592272 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.592863 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.592967 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.593013 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:11.593315 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 20:21:11.593389 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:21:11.593575 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:11.589219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:46823 (local address 127.0.116.193:59131)
0926 20:21:11.589649 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:11.589665 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:11.589796 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:11.590072 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:11.590074 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:11.590090 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:11.590213 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:11.590219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.591111 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.591115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:11.592090 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:11.592103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.592958 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.592961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:11.593165 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.593200 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 20:21:11.593279 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 20:21:11.593364 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":124,"thread_start_us":123,"threads_started":1}
I20260926 20:21:11.593684 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:11.589219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:42217 (local address 127.0.116.193:49819)
0926 20:21:11.589703 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:11.589715 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:11.589857 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:11.590197 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:11.590200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:11.590212 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:11.590332 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:11.590338 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.591145 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.591148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:11.592018 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:11.592027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.592925 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.592932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:11.593138 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.593198 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 20:21:11.593279 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 20:21:11.593364 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":83,"thread_start_us":126,"threads_started":1}
I20260926 20:21:11.594349 8292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69424bf028a45ddb2e48b000259cf14" candidate_uuid: "748991b49d7948afa8b7227a4886ad38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2caca33540084b7ba7288fbe0b70af32" is_pre_election: true
I20260926 20:21:11.594352 8434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69424bf028a45ddb2e48b000259cf14" candidate_uuid: "748991b49d7948afa8b7227a4886ad38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" is_pre_election: true
I20260926 20:21:11.594663 8292 raft_consensus.cc:2454] T a69424bf028a45ddb2e48b000259cf14 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 748991b49d7948afa8b7227a4886ad38 in term 0.
I20260926 20:21:11.594666 8434 raft_consensus.cc:2454] T a69424bf028a45ddb2e48b000259cf14 P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 748991b49d7948afa8b7227a4886ad38 in term 0.
I20260926 20:21:11.595180 8086 leader_election.cc:302] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2caca33540084b7ba7288fbe0b70af32, 748991b49d7948afa8b7227a4886ad38, af9fef83ce7b4f1faf748eff13d7fd3a; no voters:
I20260926 20:21:11.595438 8515 raft_consensus.cc:2781] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:21:11.595551 8515 raft_consensus.cc:482] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:21:11.595638 8515 raft_consensus.cc:3037] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:11.596789 8515 raft_consensus.cc:504] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } }
I20260926 20:21:11.597148 8515 leader_election.cc:288] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38 [CANDIDATE]: Term 1 election: Requested vote from peers af9fef83ce7b4f1faf748eff13d7fd3a (127.0.116.195:46823), 2caca33540084b7ba7288fbe0b70af32 (127.0.116.194:42217)
I20260926 20:21:11.597513 8292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69424bf028a45ddb2e48b000259cf14" candidate_uuid: "748991b49d7948afa8b7227a4886ad38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2caca33540084b7ba7288fbe0b70af32"
I20260926 20:21:11.597541 8434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69424bf028a45ddb2e48b000259cf14" candidate_uuid: "748991b49d7948afa8b7227a4886ad38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a"
I20260926 20:21:11.597669 8292 raft_consensus.cc:3037] T a69424bf028a45ddb2e48b000259cf14 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:11.597667 8434 raft_consensus.cc:3037] T a69424bf028a45ddb2e48b000259cf14 P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:11.598716 8292 raft_consensus.cc:2454] T a69424bf028a45ddb2e48b000259cf14 P 2caca33540084b7ba7288fbe0b70af32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 748991b49d7948afa8b7227a4886ad38 in term 1.
I20260926 20:21:11.598773 8434 raft_consensus.cc:2454] T a69424bf028a45ddb2e48b000259cf14 P af9fef83ce7b4f1faf748eff13d7fd3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 748991b49d7948afa8b7227a4886ad38 in term 1.
I20260926 20:21:11.599102 8085 leader_election.cc:302] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2caca33540084b7ba7288fbe0b70af32, 748991b49d7948afa8b7227a4886ad38, af9fef83ce7b4f1faf748eff13d7fd3a; no voters:
I20260926 20:21:11.599309 8515 raft_consensus.cc:2781] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:21:11.599596 8515 raft_consensus.cc:686] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38 [term 1 LEADER]: Becoming Leader. State: Replica: 748991b49d7948afa8b7227a4886ad38, State: Running, Role: LEADER
I20260926 20:21:11.599963 8515 consensus_queue.cc:237] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } }
I20260926 20:21:11.603296 7995 catalog_manager.cc:5495] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 748991b49d7948afa8b7227a4886ad38 (127.0.116.193). New cstate: current_term: 1 leader_uuid: "748991b49d7948afa8b7227a4886ad38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } health_report { overall_health: UNKNOWN } } }
I20260926 20:21:11.625465 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:11.620869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38182 (local address 127.0.116.193:38473)
0926 20:21:11.621036 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:11.621043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:11.621061 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:21:11.621186 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:11.621191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:11.621282 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:11.621424 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:21:11.621673 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.622569 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.623450 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.623914 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.624040 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.624305 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:21:11.624972 (+ 667us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:21:11.625107 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:11.625172 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 20:21:11.625253 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:21:11.632258 8292 raft_consensus.cc:1261] T a69424bf028a45ddb2e48b000259cf14 P 2caca33540084b7ba7288fbe0b70af32 [term 1 FOLLOWER]: Refusing update from remote peer 748991b49d7948afa8b7227a4886ad38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:21:11.632730 8434 raft_consensus.cc:1261] T a69424bf028a45ddb2e48b000259cf14 P af9fef83ce7b4f1faf748eff13d7fd3a [term 1 FOLLOWER]: Refusing update from remote peer 748991b49d7948afa8b7227a4886ad38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:21:11.633023 8519 consensus_queue.cc:1035] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:21:11.633375 8515 consensus_queue.cc:1035] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:21:11.641950 8522 mvcc.cc:205] Tried to move back new op lower bound from 7333699877398749184 to 7333699877276332032. Current Snapshot: MvccSnapshot[applied={T|T < 7333699877398749184}]
I20260926 20:21:11.642874 8524 mvcc.cc:205] Tried to move back new op lower bound from 7333699877398749184 to 7333699877276332032. Current Snapshot: MvccSnapshot[applied={T|T < 7333699877398749184}]
I20260926 20:21:11.644583 8523 mvcc.cc:205] Tried to move back new op lower bound from 7333699877398749184 to 7333699877276332032. Current Snapshot: MvccSnapshot[applied={T|T < 7333699877398749184}]
I20260926 20:21:11.656844 7997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:21:11.657058 7997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:21:11.660044 7997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9: Sending DeleteTablet for 3 replicas of tablet a69424bf028a45ddb2e48b000259cf14
I20260926 20:21:11.660991 8134 tablet_service.cc:1514] Processing DeleteTablet for tablet a69424bf028a45ddb2e48b000259cf14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:11 UTC) from {username='oryx', principal='oryx/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:38166
I20260926 20:21:11.661249 8274 tablet_service.cc:1514] Processing DeleteTablet for tablet a69424bf028a45ddb2e48b000259cf14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:11 UTC) from {username='oryx', principal='oryx/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:45314
I20260926 20:21:11.661809 8532 tablet_replica.cc:330] T a69424bf028a45ddb2e48b000259cf14 P 2caca33540084b7ba7288fbe0b70af32: stopping tablet replica
I20260926 20:21:11.662222 8532 raft_consensus.cc:2229] T a69424bf028a45ddb2e48b000259cf14 P 2caca33540084b7ba7288fbe0b70af32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:21:11.662266 8414 tablet_service.cc:1514] Processing DeleteTablet for tablet a69424bf028a45ddb2e48b000259cf14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:11 UTC) from {username='oryx', principal='oryx/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:59368
I20260926 20:21:11.662602 8532 raft_consensus.cc:2258] T a69424bf028a45ddb2e48b000259cf14 P 2caca33540084b7ba7288fbe0b70af32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:21:11.663878 8533 tablet_replica.cc:330] T a69424bf028a45ddb2e48b000259cf14 P af9fef83ce7b4f1faf748eff13d7fd3a: stopping tablet replica
I20260926 20:21:11.663880 7997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 20:21:11.664453 8533 raft_consensus.cc:2229] T a69424bf028a45ddb2e48b000259cf14 P af9fef83ce7b4f1faf748eff13d7fd3a [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 20:21:11.664443 7997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:21:11.664878 8533 raft_consensus.cc:2258] T a69424bf028a45ddb2e48b000259cf14 P af9fef83ce7b4f1faf748eff13d7fd3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:21:11.666615 8533 ts_tablet_manager.cc:1905] T a69424bf028a45ddb2e48b000259cf14 P af9fef83ce7b4f1faf748eff13d7fd3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:21:11.669533 8533 ts_tablet_manager.cc:1918] T a69424bf028a45ddb2e48b000259cf14 P af9fef83ce7b4f1faf748eff13d7fd3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:21:11.669718 8533 log.cc:1196] T a69424bf028a45ddb2e48b000259cf14 P af9fef83ce7b4f1faf748eff13d7fd3a: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/a69424bf028a45ddb2e48b000259cf14
I20260926 20:21:11.670104 8533 ts_tablet_manager.cc:1939] T a69424bf028a45ddb2e48b000259cf14 P af9fef83ce7b4f1faf748eff13d7fd3a: Deleting consensus metadata
I20260926 20:21:11.671464 7981 catalog_manager.cc:4841] TS af9fef83ce7b4f1faf748eff13d7fd3a (127.0.116.195:46823): tablet a69424bf028a45ddb2e48b000259cf14 (table test-table [id=316f48f302e146ebb3aeac59768df4e4]) successfully deleted
I20260926 20:21:11.671705 8531 tablet_replica.cc:330] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38: stopping tablet replica
I20260926 20:21:11.672034 8531 raft_consensus.cc:2229] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:21:11.672504 8531 raft_consensus.cc:2258] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:21:11.673962 8532 ts_tablet_manager.cc:1905] T a69424bf028a45ddb2e48b000259cf14 P 2caca33540084b7ba7288fbe0b70af32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:21:11.674181 8134 tablet_service.cc:1467] Processing CreateTablet for tablet f873e35959ca4c3ca5655f228682e3ae (DEFAULT_TABLE table=test-table [id=79ccf0587cb64751b9b80f88320bc9d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:21:11.674436 8134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f873e35959ca4c3ca5655f228682e3ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:11.676834 8509 tablet_bootstrap.cc:492] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38: Bootstrap starting.
I20260926 20:21:11.677086 8274 tablet_service.cc:1467] Processing CreateTablet for tablet f873e35959ca4c3ca5655f228682e3ae (DEFAULT_TABLE table=test-table [id=79ccf0587cb64751b9b80f88320bc9d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:21:11.677379 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f873e35959ca4c3ca5655f228682e3ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:11.677639 8532 ts_tablet_manager.cc:1918] T a69424bf028a45ddb2e48b000259cf14 P 2caca33540084b7ba7288fbe0b70af32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:21:11.677770 8532 log.cc:1196] T a69424bf028a45ddb2e48b000259cf14 P 2caca33540084b7ba7288fbe0b70af32: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/a69424bf028a45ddb2e48b000259cf14
I20260926 20:21:11.677942 8509 tablet_bootstrap.cc:654] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:11.678107 8532 ts_tablet_manager.cc:1939] T a69424bf028a45ddb2e48b000259cf14 P 2caca33540084b7ba7288fbe0b70af32: Deleting consensus metadata
I20260926 20:21:11.679558 8508 tablet_bootstrap.cc:492] T f873e35959ca4c3ca5655f228682e3ae P 2caca33540084b7ba7288fbe0b70af32: Bootstrap starting.
I20260926 20:21:11.680543 8508 tablet_bootstrap.cc:654] T f873e35959ca4c3ca5655f228682e3ae P 2caca33540084b7ba7288fbe0b70af32: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:11.680872 8531 ts_tablet_manager.cc:1905] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:21:11.681538 7982 catalog_manager.cc:4841] TS 2caca33540084b7ba7288fbe0b70af32 (127.0.116.194:42217): tablet a69424bf028a45ddb2e48b000259cf14 (table test-table [id=316f48f302e146ebb3aeac59768df4e4]) successfully deleted
I20260926 20:21:11.682646 8414 tablet_service.cc:1467] Processing CreateTablet for tablet f873e35959ca4c3ca5655f228682e3ae (DEFAULT_TABLE table=test-table [id=79ccf0587cb64751b9b80f88320bc9d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:21:11.682988 8414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f873e35959ca4c3ca5655f228682e3ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:11.683697 8531 ts_tablet_manager.cc:1918] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:21:11.683799 8531 log.cc:1196] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-0/wal/wals/a69424bf028a45ddb2e48b000259cf14
I20260926 20:21:11.683866 8508 tablet_bootstrap.cc:492] T f873e35959ca4c3ca5655f228682e3ae P 2caca33540084b7ba7288fbe0b70af32: No bootstrap required, opened a new log
I20260926 20:21:11.683974 8508 ts_tablet_manager.cc:1397] T f873e35959ca4c3ca5655f228682e3ae P 2caca33540084b7ba7288fbe0b70af32: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 20:21:11.684182 8531 ts_tablet_manager.cc:1939] T a69424bf028a45ddb2e48b000259cf14 P 748991b49d7948afa8b7227a4886ad38: Deleting consensus metadata
I20260926 20:21:11.684547 8508 raft_consensus.cc:348] T f873e35959ca4c3ca5655f228682e3ae P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:11.684667 8508 raft_consensus.cc:374] T f873e35959ca4c3ca5655f228682e3ae P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:11.684719 8508 raft_consensus.cc:729] T f873e35959ca4c3ca5655f228682e3ae P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2caca33540084b7ba7288fbe0b70af32, State: Initialized, Role: FOLLOWER
I20260926 20:21:11.684842 8508 consensus_queue.cc:260] T f873e35959ca4c3ca5655f228682e3ae P 2caca33540084b7ba7288fbe0b70af32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:11.685113 8508 ts_tablet_manager.cc:1428] T f873e35959ca4c3ca5655f228682e3ae P 2caca33540084b7ba7288fbe0b70af32: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:21:11.686197 8510 tablet_bootstrap.cc:492] T f873e35959ca4c3ca5655f228682e3ae P af9fef83ce7b4f1faf748eff13d7fd3a: Bootstrap starting.
I20260926 20:21:11.687158 7983 catalog_manager.cc:4841] TS 748991b49d7948afa8b7227a4886ad38 (127.0.116.193:38473): tablet a69424bf028a45ddb2e48b000259cf14 (table test-table [id=316f48f302e146ebb3aeac59768df4e4]) successfully deleted
I20260926 20:21:11.687565 8510 tablet_bootstrap.cc:654] T f873e35959ca4c3ca5655f228682e3ae P af9fef83ce7b4f1faf748eff13d7fd3a: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:11.689010 8509 tablet_bootstrap.cc:492] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38: No bootstrap required, opened a new log
I20260926 20:21:11.689116 8509 ts_tablet_manager.cc:1397] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260926 20:21:11.689740 8509 raft_consensus.cc:348] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:11.689915 8509 raft_consensus.cc:374] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:11.689949 8509 raft_consensus.cc:729] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 748991b49d7948afa8b7227a4886ad38, State: Initialized, Role: FOLLOWER
I20260926 20:21:11.690124 8509 consensus_queue.cc:260] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:11.690480 8509 ts_tablet_manager.cc:1428] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 20:21:11.693046 8510 tablet_bootstrap.cc:492] T f873e35959ca4c3ca5655f228682e3ae P af9fef83ce7b4f1faf748eff13d7fd3a: No bootstrap required, opened a new log
I20260926 20:21:11.693156 8510 ts_tablet_manager.cc:1397] T f873e35959ca4c3ca5655f228682e3ae P af9fef83ce7b4f1faf748eff13d7fd3a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260926 20:21:11.693849 8510 raft_consensus.cc:348] T f873e35959ca4c3ca5655f228682e3ae P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:11.694036 8510 raft_consensus.cc:374] T f873e35959ca4c3ca5655f228682e3ae P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:11.694087 8510 raft_consensus.cc:729] T f873e35959ca4c3ca5655f228682e3ae P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9fef83ce7b4f1faf748eff13d7fd3a, State: Initialized, Role: FOLLOWER
I20260926 20:21:11.694247 8510 consensus_queue.cc:260] T f873e35959ca4c3ca5655f228682e3ae P af9fef83ce7b4f1faf748eff13d7fd3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:11.694617 8510 ts_tablet_manager.cc:1428] T f873e35959ca4c3ca5655f228682e3ae P af9fef83ce7b4f1faf748eff13d7fd3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
W20260926 20:21:11.698972 8481 tablet.cc:2369] T f873e35959ca4c3ca5655f228682e3ae P af9fef83ce7b4f1faf748eff13d7fd3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:21:11.703999 8515 raft_consensus.cc:482] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:21:11.704134 8515 raft_consensus.cc:504] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:11.704566 8515 leader_election.cc:288] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2caca33540084b7ba7288fbe0b70af32 (127.0.116.194:42217), af9fef83ce7b4f1faf748eff13d7fd3a (127.0.116.195:46823)
I20260926 20:21:11.705168 8292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f873e35959ca4c3ca5655f228682e3ae" candidate_uuid: "748991b49d7948afa8b7227a4886ad38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2caca33540084b7ba7288fbe0b70af32" is_pre_election: true
I20260926 20:21:11.705262 8434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f873e35959ca4c3ca5655f228682e3ae" candidate_uuid: "748991b49d7948afa8b7227a4886ad38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" is_pre_election: true
I20260926 20:21:11.705353 8292 raft_consensus.cc:2454] T f873e35959ca4c3ca5655f228682e3ae P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 748991b49d7948afa8b7227a4886ad38 in term 0.
I20260926 20:21:11.705399 8434 raft_consensus.cc:2454] T f873e35959ca4c3ca5655f228682e3ae P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 748991b49d7948afa8b7227a4886ad38 in term 0.
I20260926 20:21:11.705718 8085 leader_election.cc:302] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 748991b49d7948afa8b7227a4886ad38, af9fef83ce7b4f1faf748eff13d7fd3a; no voters:
I20260926 20:21:11.705936 8515 raft_consensus.cc:2781] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:21:11.705988 8515 raft_consensus.cc:482] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:21:11.706030 8515 raft_consensus.cc:3037] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:11.707307 8515 raft_consensus.cc:504] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:11.707707 8515 leader_election.cc:288] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38 [CANDIDATE]: Term 1 election: Requested vote from peers 2caca33540084b7ba7288fbe0b70af32 (127.0.116.194:42217), af9fef83ce7b4f1faf748eff13d7fd3a (127.0.116.195:46823)
I20260926 20:21:11.708122 8292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f873e35959ca4c3ca5655f228682e3ae" candidate_uuid: "748991b49d7948afa8b7227a4886ad38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2caca33540084b7ba7288fbe0b70af32"
I20260926 20:21:11.708257 8292 raft_consensus.cc:3037] T f873e35959ca4c3ca5655f228682e3ae P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:11.709386 8292 raft_consensus.cc:2454] T f873e35959ca4c3ca5655f228682e3ae P 2caca33540084b7ba7288fbe0b70af32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 748991b49d7948afa8b7227a4886ad38 in term 1.
I20260926 20:21:11.709733 8086 leader_election.cc:302] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2caca33540084b7ba7288fbe0b70af32, 748991b49d7948afa8b7227a4886ad38; no voters:
I20260926 20:21:11.710039 8434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f873e35959ca4c3ca5655f228682e3ae" candidate_uuid: "748991b49d7948afa8b7227a4886ad38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a"
I20260926 20:21:11.710196 8434 raft_consensus.cc:3037] T f873e35959ca4c3ca5655f228682e3ae P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:11.711314 8434 raft_consensus.cc:2454] T f873e35959ca4c3ca5655f228682e3ae P af9fef83ce7b4f1faf748eff13d7fd3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 748991b49d7948afa8b7227a4886ad38 in term 1.
I20260926 20:21:11.711573 8515 raft_consensus.cc:2781] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:21:11.711687 8515 raft_consensus.cc:686] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38 [term 1 LEADER]: Becoming Leader. State: Replica: 748991b49d7948afa8b7227a4886ad38, State: Running, Role: LEADER
I20260926 20:21:11.711830 8515 consensus_queue.cc:237] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:11.713807 7997 catalog_manager.cc:5495] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 748991b49d7948afa8b7227a4886ad38 (127.0.116.193). New cstate: current_term: 1 leader_uuid: "748991b49d7948afa8b7227a4886ad38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } health_report { overall_health: UNKNOWN } } }
W20260926 20:21:11.714557 8341 tablet.cc:2369] T f873e35959ca4c3ca5655f228682e3ae P 2caca33540084b7ba7288fbe0b70af32: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:21:11.731616 8201 tablet.cc:2369] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 20:21:11 dist-test-slave-qsm9 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454070, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.254@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM
I20260926 20:21:11.758322 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:11.746060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:36149 (local address 127.0.0.1:52498)
0926 20:21:11.746508 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:11.746526 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:11.746692 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:11.747052 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:11.747057 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:11.747700 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:11.747862 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:11.747874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.749006 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.749012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:11.750160 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:11.750175 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.750800 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.750805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:11.750930 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.751755 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:11.751776 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:11.753820 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:11.756239 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:11.756252 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:11.756259 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:11.756603 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:11.756960 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:11.756963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:11.756965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:11.757066 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:11.757576 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:11.757585 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:11.757739 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 20:21:11.757869 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 20:21:11.758048 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":128,"threads_started":1}
I20260926 20:21:11.759351 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:11.746164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.0.116.254:36149)
0926 20:21:11.746325 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:11.746332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:11.746654 (+ 322us) server_negotiation.cc:407] Connection header received
0926 20:21:11.746815 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:11.746818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:11.746872 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:11.746987 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:11.748021 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.748879 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.750306 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:11.750660 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:11.750770 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:11.754021 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:11.754028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:11.754043 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:11.754078 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:11.756092 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:11.756736 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:11.756744 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:11.756749 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:11.756852 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:11.757166 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:11.757172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:11.757175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:11.757455 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:11.757566 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:11.757949 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 20:21:11.758139 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":59}
I20260926 20:21:11.764825 7997 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:52498:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 20:21:11.765686 7997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:21:11.772032 8134 tablet_service.cc:1467] Processing CreateTablet for tablet 209c1a983d4240e4b7078baeb040d711 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ad91d23098b44608b4dc45e94ad8645]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:21:11.772430 8134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209c1a983d4240e4b7078baeb040d711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:11.772423 8274 tablet_service.cc:1467] Processing CreateTablet for tablet 209c1a983d4240e4b7078baeb040d711 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ad91d23098b44608b4dc45e94ad8645]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:21:11.772840 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209c1a983d4240e4b7078baeb040d711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:11.775348 8509 tablet_bootstrap.cc:492] T 209c1a983d4240e4b7078baeb040d711 P 748991b49d7948afa8b7227a4886ad38: Bootstrap starting.
I20260926 20:21:11.776576 8509 tablet_bootstrap.cc:654] T 209c1a983d4240e4b7078baeb040d711 P 748991b49d7948afa8b7227a4886ad38: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:11.778124 8414 tablet_service.cc:1467] Processing CreateTablet for tablet 209c1a983d4240e4b7078baeb040d711 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ad91d23098b44608b4dc45e94ad8645]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:21:11.778524 8414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209c1a983d4240e4b7078baeb040d711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:11.780160 8509 tablet_bootstrap.cc:492] T 209c1a983d4240e4b7078baeb040d711 P 748991b49d7948afa8b7227a4886ad38: No bootstrap required, opened a new log
I20260926 20:21:11.780259 8509 ts_tablet_manager.cc:1397] T 209c1a983d4240e4b7078baeb040d711 P 748991b49d7948afa8b7227a4886ad38: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 20:21:11.780762 8509 raft_consensus.cc:348] T 209c1a983d4240e4b7078baeb040d711 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:11.780889 8509 raft_consensus.cc:374] T 209c1a983d4240e4b7078baeb040d711 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:11.780922 8509 raft_consensus.cc:729] T 209c1a983d4240e4b7078baeb040d711 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 748991b49d7948afa8b7227a4886ad38, State: Initialized, Role: FOLLOWER
I20260926 20:21:11.781060 8509 consensus_queue.cc:260] T 209c1a983d4240e4b7078baeb040d711 P 748991b49d7948afa8b7227a4886ad38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:11.782464 8510 tablet_bootstrap.cc:492] T 209c1a983d4240e4b7078baeb040d711 P af9fef83ce7b4f1faf748eff13d7fd3a: Bootstrap starting.
I20260926 20:21:11.782572 8515 tablet_replica.cc:439] T 209c1a983d4240e4b7078baeb040d711 P 748991b49d7948afa8b7227a4886ad38: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } } }
I20260926 20:21:11.782748 8515 tablet_replica.cc:442] T 209c1a983d4240e4b7078baeb040d711 P 748991b49d7948afa8b7227a4886ad38: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:21:11.783902 8510 tablet_bootstrap.cc:654] T 209c1a983d4240e4b7078baeb040d711 P af9fef83ce7b4f1faf748eff13d7fd3a: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:11.784175 8509 ts_tablet_manager.cc:1428] T 209c1a983d4240e4b7078baeb040d711 P 748991b49d7948afa8b7227a4886ad38: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260926 20:21:11.784668 8508 tablet_bootstrap.cc:492] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32: Bootstrap starting.
I20260926 20:21:11.785712 8508 tablet_bootstrap.cc:654] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:11.786321 8510 tablet_bootstrap.cc:492] T 209c1a983d4240e4b7078baeb040d711 P af9fef83ce7b4f1faf748eff13d7fd3a: No bootstrap required, opened a new log
I20260926 20:21:11.786409 8510 ts_tablet_manager.cc:1397] T 209c1a983d4240e4b7078baeb040d711 P af9fef83ce7b4f1faf748eff13d7fd3a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 20:21:11.786883 8510 raft_consensus.cc:348] T 209c1a983d4240e4b7078baeb040d711 P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:11.787010 8510 raft_consensus.cc:374] T 209c1a983d4240e4b7078baeb040d711 P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:11.787073 8510 raft_consensus.cc:729] T 209c1a983d4240e4b7078baeb040d711 P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9fef83ce7b4f1faf748eff13d7fd3a, State: Initialized, Role: FOLLOWER
I20260926 20:21:11.787236 8510 consensus_queue.cc:260] T 209c1a983d4240e4b7078baeb040d711 P af9fef83ce7b4f1faf748eff13d7fd3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:11.787609 8510 ts_tablet_manager.cc:1428] T 209c1a983d4240e4b7078baeb040d711 P af9fef83ce7b4f1faf748eff13d7fd3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:21:11.787667 8508 tablet_bootstrap.cc:492] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32: No bootstrap required, opened a new log
I20260926 20:21:11.787739 8508 ts_tablet_manager.cc:1397] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:21:11.788123 8508 raft_consensus.cc:348] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:11.788228 8508 raft_consensus.cc:374] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:11.788134 8516 tablet_replica.cc:439] T 209c1a983d4240e4b7078baeb040d711 P af9fef83ce7b4f1faf748eff13d7fd3a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } } }
I20260926 20:21:11.788260 8508 raft_consensus.cc:729] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2caca33540084b7ba7288fbe0b70af32, State: Initialized, Role: FOLLOWER
I20260926 20:21:11.788321 8516 tablet_replica.cc:442] T 209c1a983d4240e4b7078baeb040d711 P af9fef83ce7b4f1faf748eff13d7fd3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:21:11.788371 8508 consensus_queue.cc:260] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:11.788657 8508 ts_tablet_manager.cc:1428] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:21:11.788981 8514 tablet_replica.cc:439] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } } }
I20260926 20:21:11.789136 8514 tablet_replica.cc:442] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:21:12.100888 8514 raft_consensus.cc:482] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:21:12.101055 8514 raft_consensus.cc:504] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:12.102193 8514 leader_election.cc:288] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 748991b49d7948afa8b7227a4886ad38 (127.0.116.193:38473), af9fef83ce7b4f1faf748eff13d7fd3a (127.0.116.195:46823)
I20260926 20:21:12.106199 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:12.102662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:37419 (local address 127.0.116.193:38473)
0926 20:21:12.102828 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:12.102832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:12.103002 (+ 170us) server_negotiation.cc:407] Connection header received
0926 20:21:12.103148 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:12.103151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:12.103210 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:12.103302 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:12.103628 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.104150 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.105219 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.105629 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.105758 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:12.105825 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:12.105997 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 20:21:12.106075 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 20:21:12.106215 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:12.102515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:38473 (local address 127.0.116.194:37419)
0926 20:21:12.102835 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:12.102852 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:12.103017 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:12.103316 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:12.103319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:12.103336 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:12.103476 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:12.103482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.104265 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.104271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:12.105083 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:12.105092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.105755 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.105758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:12.105859 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:12.105881 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:21:12.105982 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 20:21:12.106055 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":100,"threads_started":1}
I20260926 20:21:12.106350 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:12.102679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:44805 (local address 127.0.116.195:46823)
0926 20:21:12.102945 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:12.102950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:12.103214 (+ 264us) server_negotiation.cc:407] Connection header received
0926 20:21:12.103378 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:12.103381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:12.103425 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:12.103556 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:12.103812 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.104333 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.105331 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.105772 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.105901 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:12.105950 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:12.106115 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 20:21:12.106192 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":80,"threads_started":1}
I20260926 20:21:12.106539 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:12.102515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:46823 (local address 127.0.116.194:44805)
0926 20:21:12.103109 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:12.103120 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:12.103260 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:12.103564 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:12.103567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:12.103582 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:12.103690 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:12.103696 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.104447 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.104452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:12.105205 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:12.105213 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.105892 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.105895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:12.105980 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:12.105999 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 20:21:12.106101 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 20:21:12.106178 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":298,"thread_start_us":49,"threads_started":1}
I20260926 20:21:12.106853 8154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209c1a983d4240e4b7078baeb040d711" candidate_uuid: "2caca33540084b7ba7288fbe0b70af32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "748991b49d7948afa8b7227a4886ad38" is_pre_election: true
I20260926 20:21:12.106956 8434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209c1a983d4240e4b7078baeb040d711" candidate_uuid: "2caca33540084b7ba7288fbe0b70af32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" is_pre_election: true
I20260926 20:21:12.107085 8434 raft_consensus.cc:2454] T 209c1a983d4240e4b7078baeb040d711 P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2caca33540084b7ba7288fbe0b70af32 in term 0.
I20260926 20:21:12.107125 8154 raft_consensus.cc:2454] T 209c1a983d4240e4b7078baeb040d711 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2caca33540084b7ba7288fbe0b70af32 in term 0.
I20260926 20:21:12.107486 8225 leader_election.cc:302] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2caca33540084b7ba7288fbe0b70af32, af9fef83ce7b4f1faf748eff13d7fd3a; no voters:
I20260926 20:21:12.107831 8514 raft_consensus.cc:2781] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:21:12.107898 8514 raft_consensus.cc:482] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:21:12.107928 8514 raft_consensus.cc:3037] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:12.108947 8514 raft_consensus.cc:504] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:12.109424 8514 leader_election.cc:288] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32 [CANDIDATE]: Term 1 election: Requested vote from peers 748991b49d7948afa8b7227a4886ad38 (127.0.116.193:38473), af9fef83ce7b4f1faf748eff13d7fd3a (127.0.116.195:46823)
I20260926 20:21:12.109798 8434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209c1a983d4240e4b7078baeb040d711" candidate_uuid: "2caca33540084b7ba7288fbe0b70af32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a"
I20260926 20:21:12.109786 8154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209c1a983d4240e4b7078baeb040d711" candidate_uuid: "2caca33540084b7ba7288fbe0b70af32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "748991b49d7948afa8b7227a4886ad38"
I20260926 20:21:12.109915 8154 raft_consensus.cc:3037] T 209c1a983d4240e4b7078baeb040d711 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:12.109915 8434 raft_consensus.cc:3037] T 209c1a983d4240e4b7078baeb040d711 P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:12.111043 8154 raft_consensus.cc:2454] T 209c1a983d4240e4b7078baeb040d711 P 748991b49d7948afa8b7227a4886ad38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2caca33540084b7ba7288fbe0b70af32 in term 1.
I20260926 20:21:12.111068 8434 raft_consensus.cc:2454] T 209c1a983d4240e4b7078baeb040d711 P af9fef83ce7b4f1faf748eff13d7fd3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2caca33540084b7ba7288fbe0b70af32 in term 1.
I20260926 20:21:12.111414 8227 leader_election.cc:302] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2caca33540084b7ba7288fbe0b70af32, 748991b49d7948afa8b7227a4886ad38; no voters:
I20260926 20:21:12.111626 8514 raft_consensus.cc:2781] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:21:12.111824 8514 raft_consensus.cc:686] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32 [term 1 LEADER]: Becoming Leader. State: Replica: 2caca33540084b7ba7288fbe0b70af32, State: Running, Role: LEADER
I20260926 20:21:12.112208 8514 consensus_queue.cc:237] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:12.113466 8550 tablet_replica.cc:439] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32: TxnStatusTablet state changed. Reason: New leader 2caca33540084b7ba7288fbe0b70af32. Latest consensus state: current_term: 1 leader_uuid: "2caca33540084b7ba7288fbe0b70af32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } } }
I20260926 20:21:12.113584 8550 tablet_replica.cc:442] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:21:12.114054 8552 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 20:21:12.114230 8552 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 20:21:12.114801 7997 catalog_manager.cc:5495] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2caca33540084b7ba7288fbe0b70af32 (127.0.116.194). New cstate: current_term: 1 leader_uuid: "2caca33540084b7ba7288fbe0b70af32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } health_report { overall_health: UNKNOWN } } }
I20260926 20:21:12.157955 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:12.154094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:42217 (local address 127.0.0.1:45322)
0926 20:21:12.154248 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:12.154265 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:12.154384 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:12.154802 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:12.154805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:12.154812 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:21:12.154927 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:12.154932 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.155755 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.155761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:12.156360 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:12.156368 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.156819 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.156822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:12.156920 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:12.156963 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:21:12.157657 (+ 694us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:21:12.157673 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 20:21:12.157742 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:21:12.157829 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 20:21:12.158170 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:12.154245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.0.116.194:42217)
0926 20:21:12.154537 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:12.154542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:12.154553 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:21:12.154616 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:12.154618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:12.154668 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:12.154793 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:21:12.155033 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.155647 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.156475 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.156709 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.156817 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:12.157133 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:21:12.157563 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:21:12.157686 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:12.157752 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 20:21:12.157805 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":96,"threads_started":1}
I20260926 20:21:12.161037 8434 raft_consensus.cc:1261] T 209c1a983d4240e4b7078baeb040d711 P af9fef83ce7b4f1faf748eff13d7fd3a [term 1 FOLLOWER]: Refusing update from remote peer 2caca33540084b7ba7288fbe0b70af32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:21:12.161273 8154 raft_consensus.cc:1261] T 209c1a983d4240e4b7078baeb040d711 P 748991b49d7948afa8b7227a4886ad38 [term 1 FOLLOWER]: Refusing update from remote peer 2caca33540084b7ba7288fbe0b70af32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:21:12.161738 8550 consensus_queue.cc:1035] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:21:12.161906 8514 consensus_queue.cc:1035] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:21:12.163414 8516 tablet_replica.cc:439] T 209c1a983d4240e4b7078baeb040d711 P af9fef83ce7b4f1faf748eff13d7fd3a: TxnStatusTablet state changed. Reason: New leader 2caca33540084b7ba7288fbe0b70af32. Latest consensus state: current_term: 1 leader_uuid: "2caca33540084b7ba7288fbe0b70af32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } } }
I20260926 20:21:12.163560 8516 tablet_replica.cc:442] T 209c1a983d4240e4b7078baeb040d711 P af9fef83ce7b4f1faf748eff13d7fd3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:21:12.163728 8515 tablet_replica.cc:439] T 209c1a983d4240e4b7078baeb040d711 P 748991b49d7948afa8b7227a4886ad38: TxnStatusTablet state changed. Reason: New leader 2caca33540084b7ba7288fbe0b70af32. Latest consensus state: current_term: 1 leader_uuid: "2caca33540084b7ba7288fbe0b70af32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } } }
I20260926 20:21:12.163843 8515 tablet_replica.cc:442] T 209c1a983d4240e4b7078baeb040d711 P 748991b49d7948afa8b7227a4886ad38: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:21:12.165807 8514 tablet_replica.cc:439] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2caca33540084b7ba7288fbe0b70af32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } } }
I20260926 20:21:12.165925 8514 tablet_replica.cc:442] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:21:12.166000 8516 tablet_replica.cc:439] T 209c1a983d4240e4b7078baeb040d711 P af9fef83ce7b4f1faf748eff13d7fd3a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2caca33540084b7ba7288fbe0b70af32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } } }
I20260926 20:21:12.166109 8516 tablet_replica.cc:442] T 209c1a983d4240e4b7078baeb040d711 P af9fef83ce7b4f1faf748eff13d7fd3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:21:12.169703 8515 tablet_replica.cc:439] T 209c1a983d4240e4b7078baeb040d711 P 748991b49d7948afa8b7227a4886ad38: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2caca33540084b7ba7288fbe0b70af32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } } }
I20260926 20:21:12.169840 8515 tablet_replica.cc:442] T 209c1a983d4240e4b7078baeb040d711 P 748991b49d7948afa8b7227a4886ad38: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:21:12.170111 8550 tablet_replica.cc:439] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2caca33540084b7ba7288fbe0b70af32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } } }
I20260926 20:21:12.170230 8550 tablet_replica.cc:442] T 209c1a983d4240e4b7078baeb040d711 P 2caca33540084b7ba7288fbe0b70af32: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:21:12.180536 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:12.176862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:56487 (local address 127.0.116.194:42217)
0926 20:21:12.177000 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:12.177004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:12.177175 (+ 171us) server_negotiation.cc:407] Connection header received
0926 20:21:12.177339 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:12.177341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:12.177385 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:12.177448 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:21:12.177695 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.178236 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.179468 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.179750 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.179834 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:12.180147 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:21:12.180216 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:21:12.180267 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:12.180381 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 20:21:12.180424 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:21:12.180616 8567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:12.176771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:42217 (local address 127.0.116.193:56487)
0926 20:21:12.177095 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:12.177110 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:12.177231 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:12.177483 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:12.177486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:12.177494 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:21:12.177606 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:12.177612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.178344 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.178348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:12.179338 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:12.179350 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.179865 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.179869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:12.179962 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:12.180018 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:21:12.180294 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:21:12.180301 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 20:21:12.180375 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 20:21:12.180463 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":116,"threads_started":1}
I20260926 20:21:12.188164 8292 raft_consensus.cc:1261] T f873e35959ca4c3ca5655f228682e3ae P 2caca33540084b7ba7288fbe0b70af32 [term 1 FOLLOWER]: Refusing update from remote peer 748991b49d7948afa8b7227a4886ad38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:21:12.188179 8434 raft_consensus.cc:1261] T f873e35959ca4c3ca5655f228682e3ae P af9fef83ce7b4f1faf748eff13d7fd3a [term 1 FOLLOWER]: Refusing update from remote peer 748991b49d7948afa8b7227a4886ad38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:21:12.188555 8519 consensus_queue.cc:1035] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:21:12.188759 8515 consensus_queue.cc:1035] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:21:12.213050 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:12.207704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:35227 (local address 127.0.116.193:38473)
0926 20:21:12.207824 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:12.207829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:12.208122 (+ 293us) server_negotiation.cc:407] Connection header received
0926 20:21:12.208232 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:12.208234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:12.208274 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:12.208333 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:21:12.208606 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.209387 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.210589 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.210808 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.210885 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:12.211203 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:21:12.211280 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:21:12.211362 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:12.212864 (+ 1502us) server_negotiation.cc:299] Negotiation successful
0926 20:21:12.212928 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:21:12.213483 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:12.207666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:38473 (local address 127.0.116.194:35227)
0926 20:21:12.208010 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:12.208027 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:12.208145 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:12.208364 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:12.208367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:12.208374 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:21:12.208484 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:12.208489 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.209526 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.209530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:12.210489 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:12.210500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.210908 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.210911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:12.210994 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:12.211045 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:21:12.211402 (+ 357us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:21:12.211411 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 20:21:12.213259 (+ 1848us) client_negotiation.cc:240] Negotiation successful
0926 20:21:12.213340 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":114,"threads_started":1}
I20260926 20:21:12.244056 7994 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:21:12.244256 7994 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:21:12.246441 7994 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9: Sending DeleteTablet for 3 replicas of tablet f873e35959ca4c3ca5655f228682e3ae
I20260926 20:21:12.247022 8134 tablet_service.cc:1514] Processing DeleteTablet for tablet f873e35959ca4c3ca5655f228682e3ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:12 UTC) from {username='oryx', principal='oryx/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:38166
I20260926 20:21:12.247160 8274 tablet_service.cc:1514] Processing DeleteTablet for tablet f873e35959ca4c3ca5655f228682e3ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:12 UTC) from {username='oryx', principal='oryx/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:45314
I20260926 20:21:12.247298 8414 tablet_service.cc:1514] Processing DeleteTablet for tablet f873e35959ca4c3ca5655f228682e3ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:12 UTC) from {username='oryx', principal='oryx/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:59368
I20260926 20:21:12.247578 8575 tablet_replica.cc:330] T f873e35959ca4c3ca5655f228682e3ae P 2caca33540084b7ba7288fbe0b70af32: stopping tablet replica
I20260926 20:21:12.247694 8574 tablet_replica.cc:330] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38: stopping tablet replica
I20260926 20:21:12.247718 8575 raft_consensus.cc:2229] T f873e35959ca4c3ca5655f228682e3ae P 2caca33540084b7ba7288fbe0b70af32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:21:12.247833 8574 raft_consensus.cc:2229] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:21:12.247860 8575 raft_consensus.cc:2258] T f873e35959ca4c3ca5655f228682e3ae P 2caca33540084b7ba7288fbe0b70af32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:21:12.248308 8574 raft_consensus.cc:2258] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:21:12.248530 8575 ts_tablet_manager.cc:1905] T f873e35959ca4c3ca5655f228682e3ae P 2caca33540084b7ba7288fbe0b70af32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:21:12.251924 8575 ts_tablet_manager.cc:1918] T f873e35959ca4c3ca5655f228682e3ae P 2caca33540084b7ba7288fbe0b70af32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:21:12.252015 8575 log.cc:1196] T f873e35959ca4c3ca5655f228682e3ae P 2caca33540084b7ba7288fbe0b70af32: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/f873e35959ca4c3ca5655f228682e3ae
I20260926 20:21:12.252274 8574 ts_tablet_manager.cc:1905] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:21:12.252296 8575 ts_tablet_manager.cc:1939] T f873e35959ca4c3ca5655f228682e3ae P 2caca33540084b7ba7288fbe0b70af32: Deleting consensus metadata
I20260926 20:21:12.252627 8576 tablet_replica.cc:330] T f873e35959ca4c3ca5655f228682e3ae P af9fef83ce7b4f1faf748eff13d7fd3a: stopping tablet replica
I20260926 20:21:12.252790 8576 raft_consensus.cc:2229] T f873e35959ca4c3ca5655f228682e3ae P af9fef83ce7b4f1faf748eff13d7fd3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:21:12.252950 8576 raft_consensus.cc:2258] T f873e35959ca4c3ca5655f228682e3ae P af9fef83ce7b4f1faf748eff13d7fd3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:21:12.253732 7982 catalog_manager.cc:4841] TS 2caca33540084b7ba7288fbe0b70af32 (127.0.116.194:42217): tablet f873e35959ca4c3ca5655f228682e3ae (table test-table [id=79ccf0587cb64751b9b80f88320bc9d3]) successfully deleted
I20260926 20:21:12.254212 8576 ts_tablet_manager.cc:1905] T f873e35959ca4c3ca5655f228682e3ae P af9fef83ce7b4f1faf748eff13d7fd3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:21:12.255929 8574 ts_tablet_manager.cc:1918] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:21:12.256031 8574 log.cc:1196] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-0/wal/wals/f873e35959ca4c3ca5655f228682e3ae
I20260926 20:21:12.256354 8574 ts_tablet_manager.cc:1939] T f873e35959ca4c3ca5655f228682e3ae P 748991b49d7948afa8b7227a4886ad38: Deleting consensus metadata
I20260926 20:21:12.257164 7983 catalog_manager.cc:4841] TS 748991b49d7948afa8b7227a4886ad38 (127.0.116.193:38473): tablet f873e35959ca4c3ca5655f228682e3ae (table test-table [id=79ccf0587cb64751b9b80f88320bc9d3]) successfully deleted
I20260926 20:21:12.257583 8576 ts_tablet_manager.cc:1918] T f873e35959ca4c3ca5655f228682e3ae P af9fef83ce7b4f1faf748eff13d7fd3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:21:12.257678 8576 log.cc:1196] T f873e35959ca4c3ca5655f228682e3ae P af9fef83ce7b4f1faf748eff13d7fd3a: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/f873e35959ca4c3ca5655f228682e3ae
I20260926 20:21:12.257932 8576 ts_tablet_manager.cc:1939] T f873e35959ca4c3ca5655f228682e3ae P af9fef83ce7b4f1faf748eff13d7fd3a: Deleting consensus metadata
I20260926 20:21:12.258785 7981 catalog_manager.cc:4841] TS af9fef83ce7b4f1faf748eff13d7fd3a (127.0.116.195:46823): tablet f873e35959ca4c3ca5655f228682e3ae (table test-table [id=79ccf0587cb64751b9b80f88320bc9d3]) successfully deleted
Sep 26 20:21:12 dist-test-slave-qsm9 krb5kdc[7952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:21:12 dist-test-slave-qsm9 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:12 dist-test-slave-qsm9 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:21:12.279255 8584 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:12.271390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52508 (local address 127.0.116.254:36149)
0926 20:21:12.271684 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:12.271688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:12.271704 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:21:12.271855 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:12.271858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:12.271904 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:12.271986 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:12.273045 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.273589 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.274963 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.275202 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.275286 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:12.279080 (+ 3794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52508: BlockingRecv error: recv got EOF from 127.0.0.1:52508 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":66,"threads_started":1}
Sep 26 20:21:12 dist-test-slave-qsm9 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM
I20260926 20:21:12.290887 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:12.281037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52522 (local address 127.0.116.254:36149)
0926 20:21:12.281168 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:12.281171 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:12.281249 (+ 78us) server_negotiation.cc:407] Connection header received
0926 20:21:12.281401 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:12.281403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:12.281465 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:12.281536 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:12.282364 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.283039 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.284412 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.284750 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.284877 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:12.287438 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:12.287442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:12.287444 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:12.287473 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:12.289187 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:12.289625 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:12.289630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:12.289631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:12.289718 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:12.290003 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:12.290006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:12.290008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:12.290205 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:12.290344 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:12.290585 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 20:21:12.290713 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:21:12.293771 7994 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:21:12.294243 7994 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:21:12.299093 8274 tablet_service.cc:1467] Processing CreateTablet for tablet c595197a112f42c69a9e6597d1bcc556 (DEFAULT_TABLE table=test-table [id=fa92a211e5b449d68d9a6ec85cac72aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:21:12.299120 8134 tablet_service.cc:1467] Processing CreateTablet for tablet c595197a112f42c69a9e6597d1bcc556 (DEFAULT_TABLE table=test-table [id=fa92a211e5b449d68d9a6ec85cac72aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:21:12.299401 8134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c595197a112f42c69a9e6597d1bcc556. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:12.299412 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c595197a112f42c69a9e6597d1bcc556. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:12.301579 8590 tablet_bootstrap.cc:492] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38: Bootstrap starting.
I20260926 20:21:12.301510 8414 tablet_service.cc:1467] Processing CreateTablet for tablet c595197a112f42c69a9e6597d1bcc556 (DEFAULT_TABLE table=test-table [id=fa92a211e5b449d68d9a6ec85cac72aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:21:12.301818 8414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c595197a112f42c69a9e6597d1bcc556. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:12.302078 8591 tablet_bootstrap.cc:492] T c595197a112f42c69a9e6597d1bcc556 P 2caca33540084b7ba7288fbe0b70af32: Bootstrap starting.
I20260926 20:21:12.302430 8590 tablet_bootstrap.cc:654] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:12.302948 8591 tablet_bootstrap.cc:654] T c595197a112f42c69a9e6597d1bcc556 P 2caca33540084b7ba7288fbe0b70af32: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:12.303712 8590 tablet_bootstrap.cc:492] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38: No bootstrap required, opened a new log
I20260926 20:21:12.303823 8590 ts_tablet_manager.cc:1397] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:21:12.304250 8591 tablet_bootstrap.cc:492] T c595197a112f42c69a9e6597d1bcc556 P 2caca33540084b7ba7288fbe0b70af32: No bootstrap required, opened a new log
I20260926 20:21:12.304275 8590 raft_consensus.cc:348] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:12.304354 8591 ts_tablet_manager.cc:1397] T c595197a112f42c69a9e6597d1bcc556 P 2caca33540084b7ba7288fbe0b70af32: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:21:12.304411 8590 raft_consensus.cc:374] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:12.304473 8590 raft_consensus.cc:729] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 748991b49d7948afa8b7227a4886ad38, State: Initialized, Role: FOLLOWER
I20260926 20:21:12.304618 8590 consensus_queue.cc:260] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:12.304798 8591 raft_consensus.cc:348] T c595197a112f42c69a9e6597d1bcc556 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:12.304910 8591 raft_consensus.cc:374] T c595197a112f42c69a9e6597d1bcc556 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:12.304941 8591 raft_consensus.cc:729] T c595197a112f42c69a9e6597d1bcc556 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2caca33540084b7ba7288fbe0b70af32, State: Initialized, Role: FOLLOWER
I20260926 20:21:12.304962 8590 ts_tablet_manager.cc:1428] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 20:21:12.305080 8591 consensus_queue.cc:260] T c595197a112f42c69a9e6597d1bcc556 P 2caca33540084b7ba7288fbe0b70af32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:12.305562 8594 tablet_bootstrap.cc:492] T c595197a112f42c69a9e6597d1bcc556 P af9fef83ce7b4f1faf748eff13d7fd3a: Bootstrap starting.
I20260926 20:21:12.305995 8591 ts_tablet_manager.cc:1428] T c595197a112f42c69a9e6597d1bcc556 P 2caca33540084b7ba7288fbe0b70af32: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:21:12.306866 8594 tablet_bootstrap.cc:654] T c595197a112f42c69a9e6597d1bcc556 P af9fef83ce7b4f1faf748eff13d7fd3a: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:12.308501 8594 tablet_bootstrap.cc:492] T c595197a112f42c69a9e6597d1bcc556 P af9fef83ce7b4f1faf748eff13d7fd3a: No bootstrap required, opened a new log
I20260926 20:21:12.308575 8594 ts_tablet_manager.cc:1397] T c595197a112f42c69a9e6597d1bcc556 P af9fef83ce7b4f1faf748eff13d7fd3a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:21:12.308948 8594 raft_consensus.cc:348] T c595197a112f42c69a9e6597d1bcc556 P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:12.309048 8594 raft_consensus.cc:374] T c595197a112f42c69a9e6597d1bcc556 P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:12.309106 8594 raft_consensus.cc:729] T c595197a112f42c69a9e6597d1bcc556 P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9fef83ce7b4f1faf748eff13d7fd3a, State: Initialized, Role: FOLLOWER
I20260926 20:21:12.309259 8594 consensus_queue.cc:260] T c595197a112f42c69a9e6597d1bcc556 P af9fef83ce7b4f1faf748eff13d7fd3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:12.309504 8594 ts_tablet_manager.cc:1428] T c595197a112f42c69a9e6597d1bcc556 P af9fef83ce7b4f1faf748eff13d7fd3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 20:21:12.374418 8570 raft_consensus.cc:482] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:21:12.374567 8570 raft_consensus.cc:504] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:12.375113 8570 leader_election.cc:288] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2caca33540084b7ba7288fbe0b70af32 (127.0.116.194:42217), af9fef83ce7b4f1faf748eff13d7fd3a (127.0.116.195:46823)
I20260926 20:21:12.375619 8292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c595197a112f42c69a9e6597d1bcc556" candidate_uuid: "748991b49d7948afa8b7227a4886ad38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2caca33540084b7ba7288fbe0b70af32" is_pre_election: true
I20260926 20:21:12.375813 8292 raft_consensus.cc:2454] T c595197a112f42c69a9e6597d1bcc556 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 748991b49d7948afa8b7227a4886ad38 in term 0.
I20260926 20:21:12.375797 8434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c595197a112f42c69a9e6597d1bcc556" candidate_uuid: "748991b49d7948afa8b7227a4886ad38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" is_pre_election: true
I20260926 20:21:12.375934 8434 raft_consensus.cc:2454] T c595197a112f42c69a9e6597d1bcc556 P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 748991b49d7948afa8b7227a4886ad38 in term 0.
I20260926 20:21:12.376159 8086 leader_election.cc:302] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2caca33540084b7ba7288fbe0b70af32, 748991b49d7948afa8b7227a4886ad38; no voters:
I20260926 20:21:12.376408 8570 raft_consensus.cc:2781] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:21:12.376477 8570 raft_consensus.cc:482] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:21:12.376504 8570 raft_consensus.cc:3037] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:12.377602 8570 raft_consensus.cc:504] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:12.377945 8570 leader_election.cc:288] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38 [CANDIDATE]: Term 1 election: Requested vote from peers 2caca33540084b7ba7288fbe0b70af32 (127.0.116.194:42217), af9fef83ce7b4f1faf748eff13d7fd3a (127.0.116.195:46823)
I20260926 20:21:12.378367 8292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c595197a112f42c69a9e6597d1bcc556" candidate_uuid: "748991b49d7948afa8b7227a4886ad38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2caca33540084b7ba7288fbe0b70af32"
I20260926 20:21:12.378381 8434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c595197a112f42c69a9e6597d1bcc556" candidate_uuid: "748991b49d7948afa8b7227a4886ad38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a"
I20260926 20:21:12.378499 8434 raft_consensus.cc:3037] T c595197a112f42c69a9e6597d1bcc556 P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:12.378499 8292 raft_consensus.cc:3037] T c595197a112f42c69a9e6597d1bcc556 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:12.379712 8434 raft_consensus.cc:2454] T c595197a112f42c69a9e6597d1bcc556 P af9fef83ce7b4f1faf748eff13d7fd3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 748991b49d7948afa8b7227a4886ad38 in term 1.
I20260926 20:21:12.379712 8292 raft_consensus.cc:2454] T c595197a112f42c69a9e6597d1bcc556 P 2caca33540084b7ba7288fbe0b70af32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 748991b49d7948afa8b7227a4886ad38 in term 1.
I20260926 20:21:12.380018 8086 leader_election.cc:302] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2caca33540084b7ba7288fbe0b70af32, 748991b49d7948afa8b7227a4886ad38; no voters:
I20260926 20:21:12.380228 8570 raft_consensus.cc:2781] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:21:12.380327 8570 raft_consensus.cc:686] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38 [term 1 LEADER]: Becoming Leader. State: Replica: 748991b49d7948afa8b7227a4886ad38, State: Running, Role: LEADER
I20260926 20:21:12.380502 8570 consensus_queue.cc:237] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:12.382269 7994 catalog_manager.cc:5495] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 748991b49d7948afa8b7227a4886ad38 (127.0.116.193). New cstate: current_term: 1 leader_uuid: "748991b49d7948afa8b7227a4886ad38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } health_report { overall_health: UNKNOWN } } }
I20260926 20:21:12.407308 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:12.404371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38188 (local address 127.0.116.193:38473)
0926 20:21:12.404512 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:12.404515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:12.404528 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:21:12.404621 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:12.404623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:12.404673 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:12.404741 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:21:12.404954 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.405457 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.406283 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.406493 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.406570 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:12.406891 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:21:12.406962 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:21:12.407019 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:12.407143 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 20:21:12.407196 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:21:12.409399 8292 raft_consensus.cc:1261] T c595197a112f42c69a9e6597d1bcc556 P 2caca33540084b7ba7288fbe0b70af32 [term 1 FOLLOWER]: Refusing update from remote peer 748991b49d7948afa8b7227a4886ad38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:21:12.409399 8434 raft_consensus.cc:1261] T c595197a112f42c69a9e6597d1bcc556 P af9fef83ce7b4f1faf748eff13d7fd3a [term 1 FOLLOWER]: Refusing update from remote peer 748991b49d7948afa8b7227a4886ad38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:21:12.409881 8570 consensus_queue.cc:1035] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:21:12.410063 8515 consensus_queue.cc:1035] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:21:12.417548 7997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:21:12.417749 7997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:21:12.420159 7997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9: Sending DeleteTablet for 3 replicas of tablet c595197a112f42c69a9e6597d1bcc556
I20260926 20:21:12.420660 8134 tablet_service.cc:1514] Processing DeleteTablet for tablet c595197a112f42c69a9e6597d1bcc556 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:12 UTC) from {username='oryx', principal='oryx/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:38166
I20260926 20:21:12.420732 8414 tablet_service.cc:1514] Processing DeleteTablet for tablet c595197a112f42c69a9e6597d1bcc556 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:12 UTC) from {username='oryx', principal='oryx/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:59368
I20260926 20:21:12.420851 8574 tablet_replica.cc:330] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38: stopping tablet replica
I20260926 20:21:12.420835 8274 tablet_service.cc:1514] Processing DeleteTablet for tablet c595197a112f42c69a9e6597d1bcc556 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:12 UTC) from {username='oryx', principal='oryx/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:45314
I20260926 20:21:12.420959 8574 raft_consensus.cc:2229] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:21:12.421185 8574 raft_consensus.cc:2258] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:21:12.421128 8576 tablet_replica.cc:330] T c595197a112f42c69a9e6597d1bcc556 P af9fef83ce7b4f1faf748eff13d7fd3a: stopping tablet replica
I20260926 20:21:12.421212 8575 tablet_replica.cc:330] T c595197a112f42c69a9e6597d1bcc556 P 2caca33540084b7ba7288fbe0b70af32: stopping tablet replica
I20260926 20:21:12.421128 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet c595197a112f42c69a9e6597d1bcc556 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:12 UTC) from {username='oryx', principal='oryx/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:59368
I20260926 20:21:12.421340 8575 raft_consensus.cc:2229] T c595197a112f42c69a9e6597d1bcc556 P 2caca33540084b7ba7288fbe0b70af32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:21:12.421348 8576 raft_consensus.cc:2229] T c595197a112f42c69a9e6597d1bcc556 P af9fef83ce7b4f1faf748eff13d7fd3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:21:12.421466 8575 raft_consensus.cc:2258] T c595197a112f42c69a9e6597d1bcc556 P 2caca33540084b7ba7288fbe0b70af32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:21:12.421483 8576 raft_consensus.cc:2258] T c595197a112f42c69a9e6597d1bcc556 P af9fef83ce7b4f1faf748eff13d7fd3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:21:12.422067 8574 ts_tablet_manager.cc:1905] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:21:12.422200 8575 ts_tablet_manager.cc:1905] T c595197a112f42c69a9e6597d1bcc556 P 2caca33540084b7ba7288fbe0b70af32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:21:12.424371 8574 ts_tablet_manager.cc:1918] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:21:12.424463 8574 log.cc:1196] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-0/wal/wals/c595197a112f42c69a9e6597d1bcc556
I20260926 20:21:12.424508 8575 ts_tablet_manager.cc:1918] T c595197a112f42c69a9e6597d1bcc556 P 2caca33540084b7ba7288fbe0b70af32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:21:12.424589 8575 log.cc:1196] T c595197a112f42c69a9e6597d1bcc556 P 2caca33540084b7ba7288fbe0b70af32: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/c595197a112f42c69a9e6597d1bcc556
I20260926 20:21:12.424568 8576 ts_tablet_manager.cc:1905] T c595197a112f42c69a9e6597d1bcc556 P af9fef83ce7b4f1faf748eff13d7fd3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:21:12.424741 8574 ts_tablet_manager.cc:1939] T c595197a112f42c69a9e6597d1bcc556 P 748991b49d7948afa8b7227a4886ad38: Deleting consensus metadata
I20260926 20:21:12.424853 8575 ts_tablet_manager.cc:1939] T c595197a112f42c69a9e6597d1bcc556 P 2caca33540084b7ba7288fbe0b70af32: Deleting consensus metadata
I20260926 20:21:12.425448 7983 catalog_manager.cc:4841] TS 748991b49d7948afa8b7227a4886ad38 (127.0.116.193:38473): tablet c595197a112f42c69a9e6597d1bcc556 (table test-table [id=fa92a211e5b449d68d9a6ec85cac72aa]) successfully deleted
I20260926 20:21:12.426222 7994 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:21:12.426704 7994 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:21:12.426808 8576 ts_tablet_manager.cc:1918] T c595197a112f42c69a9e6597d1bcc556 P af9fef83ce7b4f1faf748eff13d7fd3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:21:12.426888 8576 log.cc:1196] T c595197a112f42c69a9e6597d1bcc556 P af9fef83ce7b4f1faf748eff13d7fd3a: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/c595197a112f42c69a9e6597d1bcc556
I20260926 20:21:12.427191 8576 ts_tablet_manager.cc:1939] T c595197a112f42c69a9e6597d1bcc556 P af9fef83ce7b4f1faf748eff13d7fd3a: Deleting consensus metadata
I20260926 20:21:12.427683 7982 catalog_manager.cc:4841] TS 2caca33540084b7ba7288fbe0b70af32 (127.0.116.194:42217): tablet c595197a112f42c69a9e6597d1bcc556 (table test-table [id=fa92a211e5b449d68d9a6ec85cac72aa]) successfully deleted
I20260926 20:21:12.427876 7981 catalog_manager.cc:4841] TS af9fef83ce7b4f1faf748eff13d7fd3a (127.0.116.195:46823): tablet c595197a112f42c69a9e6597d1bcc556 (table test-table [id=fa92a211e5b449d68d9a6ec85cac72aa]) successfully deleted
W20260926 20:21:12.428248 7981 catalog_manager.cc:4808] TS af9fef83ce7b4f1faf748eff13d7fd3a (127.0.116.195:46823): delete failed for tablet c595197a112f42c69a9e6597d1bcc556 because the tablet was not found. No further retry: Not found: Tablet not found: c595197a112f42c69a9e6597d1bcc556
I20260926 20:21:12.432299 8134 tablet_service.cc:1467] Processing CreateTablet for tablet 65d13876e8ed4fb797ea3d24813d68d5 (DEFAULT_TABLE table=test-table [id=0bae570ff5ea4d3ebdc2eb8676e0325b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:21:12.432329 8274 tablet_service.cc:1467] Processing CreateTablet for tablet 65d13876e8ed4fb797ea3d24813d68d5 (DEFAULT_TABLE table=test-table [id=0bae570ff5ea4d3ebdc2eb8676e0325b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:21:12.432437 8413 tablet_service.cc:1467] Processing CreateTablet for tablet 65d13876e8ed4fb797ea3d24813d68d5 (DEFAULT_TABLE table=test-table [id=0bae570ff5ea4d3ebdc2eb8676e0325b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:21:12.432636 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d13876e8ed4fb797ea3d24813d68d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:12.432638 8134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d13876e8ed4fb797ea3d24813d68d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:12.432674 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d13876e8ed4fb797ea3d24813d68d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:12.434628 8590 tablet_bootstrap.cc:492] T 65d13876e8ed4fb797ea3d24813d68d5 P 748991b49d7948afa8b7227a4886ad38: Bootstrap starting.
I20260926 20:21:12.435009 8594 tablet_bootstrap.cc:492] T 65d13876e8ed4fb797ea3d24813d68d5 P af9fef83ce7b4f1faf748eff13d7fd3a: Bootstrap starting.
I20260926 20:21:12.435659 8590 tablet_bootstrap.cc:654] T 65d13876e8ed4fb797ea3d24813d68d5 P 748991b49d7948afa8b7227a4886ad38: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:12.435921 8594 tablet_bootstrap.cc:654] T 65d13876e8ed4fb797ea3d24813d68d5 P af9fef83ce7b4f1faf748eff13d7fd3a: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:12.436262 8591 tablet_bootstrap.cc:492] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32: Bootstrap starting.
I20260926 20:21:12.437081 8594 tablet_bootstrap.cc:492] T 65d13876e8ed4fb797ea3d24813d68d5 P af9fef83ce7b4f1faf748eff13d7fd3a: No bootstrap required, opened a new log
I20260926 20:21:12.437175 8594 ts_tablet_manager.cc:1397] T 65d13876e8ed4fb797ea3d24813d68d5 P af9fef83ce7b4f1faf748eff13d7fd3a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:21:12.437197 8591 tablet_bootstrap.cc:654] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:12.437698 8594 raft_consensus.cc:348] T 65d13876e8ed4fb797ea3d24813d68d5 P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:12.437819 8594 raft_consensus.cc:374] T 65d13876e8ed4fb797ea3d24813d68d5 P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:12.437866 8594 raft_consensus.cc:729] T 65d13876e8ed4fb797ea3d24813d68d5 P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9fef83ce7b4f1faf748eff13d7fd3a, State: Initialized, Role: FOLLOWER
I20260926 20:21:12.438011 8594 consensus_queue.cc:260] T 65d13876e8ed4fb797ea3d24813d68d5 P af9fef83ce7b4f1faf748eff13d7fd3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:12.438191 8590 tablet_bootstrap.cc:492] T 65d13876e8ed4fb797ea3d24813d68d5 P 748991b49d7948afa8b7227a4886ad38: No bootstrap required, opened a new log
I20260926 20:21:12.438271 8590 ts_tablet_manager.cc:1397] T 65d13876e8ed4fb797ea3d24813d68d5 P 748991b49d7948afa8b7227a4886ad38: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260926 20:21:12.438305 8594 ts_tablet_manager.cc:1428] T 65d13876e8ed4fb797ea3d24813d68d5 P af9fef83ce7b4f1faf748eff13d7fd3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 20:21:12.438687 8590 raft_consensus.cc:348] T 65d13876e8ed4fb797ea3d24813d68d5 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:12.438812 8590 raft_consensus.cc:374] T 65d13876e8ed4fb797ea3d24813d68d5 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:12.438848 8590 raft_consensus.cc:729] T 65d13876e8ed4fb797ea3d24813d68d5 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 748991b49d7948afa8b7227a4886ad38, State: Initialized, Role: FOLLOWER
I20260926 20:21:12.439000 8590 consensus_queue.cc:260] T 65d13876e8ed4fb797ea3d24813d68d5 P 748991b49d7948afa8b7227a4886ad38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:12.439296 8590 ts_tablet_manager.cc:1428] T 65d13876e8ed4fb797ea3d24813d68d5 P 748991b49d7948afa8b7227a4886ad38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 20:21:12.439917 8591 tablet_bootstrap.cc:492] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32: No bootstrap required, opened a new log
I20260926 20:21:12.440001 8591 ts_tablet_manager.cc:1397] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 20:21:12.440418 8591 raft_consensus.cc:348] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:12.440559 8591 raft_consensus.cc:374] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:12.440603 8591 raft_consensus.cc:729] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2caca33540084b7ba7288fbe0b70af32, State: Initialized, Role: FOLLOWER
I20260926 20:21:12.440747 8591 consensus_queue.cc:260] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:12.441030 8591 ts_tablet_manager.cc:1428] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:21:12.466480 8514 raft_consensus.cc:482] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:21:12.466610 8514 raft_consensus.cc:504] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:12.467067 8514 leader_election.cc:288] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 748991b49d7948afa8b7227a4886ad38 (127.0.116.193:38473), af9fef83ce7b4f1faf748eff13d7fd3a (127.0.116.195:46823)
I20260926 20:21:12.467447 8434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d13876e8ed4fb797ea3d24813d68d5" candidate_uuid: "2caca33540084b7ba7288fbe0b70af32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" is_pre_election: true
I20260926 20:21:12.467455 8154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d13876e8ed4fb797ea3d24813d68d5" candidate_uuid: "2caca33540084b7ba7288fbe0b70af32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "748991b49d7948afa8b7227a4886ad38" is_pre_election: true
I20260926 20:21:12.467651 8154 raft_consensus.cc:2454] T 65d13876e8ed4fb797ea3d24813d68d5 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2caca33540084b7ba7288fbe0b70af32 in term 0.
I20260926 20:21:12.467653 8434 raft_consensus.cc:2454] T 65d13876e8ed4fb797ea3d24813d68d5 P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2caca33540084b7ba7288fbe0b70af32 in term 0.
I20260926 20:21:12.467959 8225 leader_election.cc:302] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2caca33540084b7ba7288fbe0b70af32, af9fef83ce7b4f1faf748eff13d7fd3a; no voters:
I20260926 20:21:12.468146 8514 raft_consensus.cc:2781] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:21:12.468231 8514 raft_consensus.cc:482] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:21:12.468287 8514 raft_consensus.cc:3037] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:12.469245 8514 raft_consensus.cc:504] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:12.469590 8514 leader_election.cc:288] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32 [CANDIDATE]: Term 1 election: Requested vote from peers 748991b49d7948afa8b7227a4886ad38 (127.0.116.193:38473), af9fef83ce7b4f1faf748eff13d7fd3a (127.0.116.195:46823)
I20260926 20:21:12.469950 8154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d13876e8ed4fb797ea3d24813d68d5" candidate_uuid: "2caca33540084b7ba7288fbe0b70af32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "748991b49d7948afa8b7227a4886ad38"
I20260926 20:21:12.469959 8434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d13876e8ed4fb797ea3d24813d68d5" candidate_uuid: "2caca33540084b7ba7288fbe0b70af32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a"
I20260926 20:21:12.470067 8154 raft_consensus.cc:3037] T 65d13876e8ed4fb797ea3d24813d68d5 P 748991b49d7948afa8b7227a4886ad38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:12.470088 8434 raft_consensus.cc:3037] T 65d13876e8ed4fb797ea3d24813d68d5 P af9fef83ce7b4f1faf748eff13d7fd3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:12.470968 8434 raft_consensus.cc:2454] T 65d13876e8ed4fb797ea3d24813d68d5 P af9fef83ce7b4f1faf748eff13d7fd3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2caca33540084b7ba7288fbe0b70af32 in term 1.
I20260926 20:21:12.470995 8154 raft_consensus.cc:2454] T 65d13876e8ed4fb797ea3d24813d68d5 P 748991b49d7948afa8b7227a4886ad38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2caca33540084b7ba7288fbe0b70af32 in term 1.
I20260926 20:21:12.471287 8225 leader_election.cc:302] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2caca33540084b7ba7288fbe0b70af32, af9fef83ce7b4f1faf748eff13d7fd3a; no voters:
I20260926 20:21:12.471531 8514 raft_consensus.cc:2781] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:21:12.471661 8514 raft_consensus.cc:686] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32 [term 1 LEADER]: Becoming Leader. State: Replica: 2caca33540084b7ba7288fbe0b70af32, State: Running, Role: LEADER
I20260926 20:21:12.471805 8514 consensus_queue.cc:237] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } }
I20260926 20:21:12.473572 7997 catalog_manager.cc:5495] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2caca33540084b7ba7288fbe0b70af32 (127.0.116.194). New cstate: current_term: 1 leader_uuid: "2caca33540084b7ba7288fbe0b70af32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2caca33540084b7ba7288fbe0b70af32" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 } health_report { overall_health: UNKNOWN } } }
I20260926 20:21:12.498457 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:12.495356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45328 (local address 127.0.116.194:42217)
0926 20:21:12.495484 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:12.495515 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:12.495528 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:21:12.495576 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:12.495578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:12.495622 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:12.495706 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:21:12.495958 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.496506 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.497342 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.497572 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.497689 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:12.497990 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:21:12.498070 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:21:12.498154 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:12.498250 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 20:21:12.498303 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:21:12.507395 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:12.504425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:42217 (local address 127.0.116.194:35809)
0926 20:21:12.504598 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:12.504611 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:12.504694 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:12.504903 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:12.504905 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:12.504910 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:21:12.504996 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:12.505000 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.505694 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.505697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:12.506340 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:12.506348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.506806 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.506809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:12.506873 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:12.506898 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:21:12.507149 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:21:12.507154 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:21:12.507202 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 20:21:12.507279 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 20:21:12.507395 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:12.504500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:35809 (local address 127.0.116.194:42217)
0926 20:21:12.504624 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:12.504627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:12.504657 (+ 30us) server_negotiation.cc:407] Connection header received
0926 20:21:12.504777 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:12.504780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:12.504819 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:12.504880 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:21:12.505080 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.505569 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.506455 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:12.506673 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:12.506752 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:12.507003 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:21:12.507073 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:21:12.507124 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:12.507216 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 20:21:12.507258 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:21:12.513932 8154 raft_consensus.cc:1261] T 65d13876e8ed4fb797ea3d24813d68d5 P 748991b49d7948afa8b7227a4886ad38 [term 1 FOLLOWER]: Refusing update from remote peer 2caca33540084b7ba7288fbe0b70af32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:21:12.513973 8434 raft_consensus.cc:1261] T 65d13876e8ed4fb797ea3d24813d68d5 P af9fef83ce7b4f1faf748eff13d7fd3a [term 1 FOLLOWER]: Refusing update from remote peer 2caca33540084b7ba7288fbe0b70af32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:21:12.514393 8562 consensus_queue.cc:1035] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "748991b49d7948afa8b7227a4886ad38" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 38473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:21:12.514561 8514 consensus_queue.cc:1035] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af9fef83ce7b4f1faf748eff13d7fd3a" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:21:12.548864 7994 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:21:12.549043 7994 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:21:12.551218 7994 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5832e6267154118849270dd5bfccfc9: Sending DeleteTablet for 3 replicas of tablet 65d13876e8ed4fb797ea3d24813d68d5
I20260926 20:21:12.551903 8134 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d13876e8ed4fb797ea3d24813d68d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:12 UTC) from {username='oryx', principal='oryx/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:38166
I20260926 20:21:12.552138 8274 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d13876e8ed4fb797ea3d24813d68d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:12 UTC) from {username='oryx', principal='oryx/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:45314
I20260926 20:21:12.552157 8574 tablet_replica.cc:330] T 65d13876e8ed4fb797ea3d24813d68d5 P 748991b49d7948afa8b7227a4886ad38: stopping tablet replica
I20260926 20:21:12.552161 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d13876e8ed4fb797ea3d24813d68d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:12 UTC) from {username='oryx', principal='oryx/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:59368
I20260926 20:21:12.552335 8574 raft_consensus.cc:2229] T 65d13876e8ed4fb797ea3d24813d68d5 P 748991b49d7948afa8b7227a4886ad38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:21:12.552357 8576 tablet_replica.cc:330] T 65d13876e8ed4fb797ea3d24813d68d5 P af9fef83ce7b4f1faf748eff13d7fd3a: stopping tablet replica
I20260926 20:21:12.552450 8576 raft_consensus.cc:2229] T 65d13876e8ed4fb797ea3d24813d68d5 P af9fef83ce7b4f1faf748eff13d7fd3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:21:12.552464 8574 raft_consensus.cc:2258] T 65d13876e8ed4fb797ea3d24813d68d5 P 748991b49d7948afa8b7227a4886ad38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:21:12.552490 8575 tablet_replica.cc:330] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32: stopping tablet replica
I20260926 20:21:12.552577 8576 raft_consensus.cc:2258] T 65d13876e8ed4fb797ea3d24813d68d5 P af9fef83ce7b4f1faf748eff13d7fd3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:21:12.552605 8575 raft_consensus.cc:2229] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:21:12.552911 8575 raft_consensus.cc:2258] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:21:12.553349 8576 ts_tablet_manager.cc:1905] T 65d13876e8ed4fb797ea3d24813d68d5 P af9fef83ce7b4f1faf748eff13d7fd3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:21:12.553580 8575 ts_tablet_manager.cc:1905] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:21:12.554323 8574 ts_tablet_manager.cc:1905] T 65d13876e8ed4fb797ea3d24813d68d5 P 748991b49d7948afa8b7227a4886ad38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:21:12.556061 8576 ts_tablet_manager.cc:1918] T 65d13876e8ed4fb797ea3d24813d68d5 P af9fef83ce7b4f1faf748eff13d7fd3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:21:12.556089 8575 ts_tablet_manager.cc:1918] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:21:12.556131 8576 log.cc:1196] T 65d13876e8ed4fb797ea3d24813d68d5 P af9fef83ce7b4f1faf748eff13d7fd3a: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/65d13876e8ed4fb797ea3d24813d68d5
I20260926 20:21:12.556156 8575 log.cc:1196] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/65d13876e8ed4fb797ea3d24813d68d5
I20260926 20:21:12.556401 8576 ts_tablet_manager.cc:1939] T 65d13876e8ed4fb797ea3d24813d68d5 P af9fef83ce7b4f1faf748eff13d7fd3a: Deleting consensus metadata
I20260926 20:21:12.556437 8575 ts_tablet_manager.cc:1939] T 65d13876e8ed4fb797ea3d24813d68d5 P 2caca33540084b7ba7288fbe0b70af32: Deleting consensus metadata
I20260926 20:21:12.557080 8574 ts_tablet_manager.cc:1918] T 65d13876e8ed4fb797ea3d24813d68d5 P 748991b49d7948afa8b7227a4886ad38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:21:12.557161 8574 log.cc:1196] T 65d13876e8ed4fb797ea3d24813d68d5 P 748991b49d7948afa8b7227a4886ad38: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790454033974695-467-0/minicluster-data/ts-0/wal/wals/65d13876e8ed4fb797ea3d24813d68d5
I20260926 20:21:12.557163 7982 catalog_manager.cc:4841] TS 2caca33540084b7ba7288fbe0b70af32 (127.0.116.194:42217): tablet 65d13876e8ed4fb797ea3d24813d68d5 (table test-table [id=0bae570ff5ea4d3ebdc2eb8676e0325b]) successfully deleted
I20260926 20:21:12.557178 7981 catalog_manager.cc:4841] TS af9fef83ce7b4f1faf748eff13d7fd3a (127.0.116.195:46823): tablet 65d13876e8ed4fb797ea3d24813d68d5 (table test-table [id=0bae570ff5ea4d3ebdc2eb8676e0325b]) successfully deleted
I20260926 20:21:12.557425 8574 ts_tablet_manager.cc:1939] T 65d13876e8ed4fb797ea3d24813d68d5 P 748991b49d7948afa8b7227a4886ad38: Deleting consensus metadata
I20260926 20:21:12.558066 7983 catalog_manager.cc:4841] TS 748991b49d7948afa8b7227a4886ad38 (127.0.116.193:38473): tablet 65d13876e8ed4fb797ea3d24813d68d5 (table test-table [id=0bae570ff5ea4d3ebdc2eb8676e0325b]) successfully deleted
I20260926 20:21:12.558145 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 8069
I20260926 20:21:12.565917 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 8209
I20260926 20:21:12.572793 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 8349
I20260926 20:21:12.578405 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 7965
2026-09-26T20:21:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3258 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:21:12 dist-test-slave-qsm9 krb5kdc[8610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:21:12 dist-test-slave-qsm9 krb5kdc[8610](info): set up 2 sockets
Sep 26 20:21:12 dist-test-slave-qsm9 krb5kdc[8610](info): commencing operation
krb5kdc: starting...
W20260926 20:21:13.640662 467 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:21:13 dist-test-slave-qsm9 krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:21:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:21:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:21:13.772208 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:36141
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:43935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:36141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:13.884478 8623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:13.884837 8623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:13.884924 8623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:13.889200 8623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:21:13.889295 8623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:13.889331 8623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:13.889371 8623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:21:13.889403 8623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:21:13.894958 8623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:43935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:36141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:36141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:13.896209 8623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:13.897275 8623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:13.906145 8631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:13.906260 8628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:13.906311 8629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:13.906605 8623 server_base.cc:1034] running on GCE node
I20260926 20:21:13.907265 8623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:13.908505 8623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:13.909711 8623 hybrid_clock.cc:648] HybridClock initialized: now 1790454073909696 us; error 28 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T20:21:13Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1340 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:21:13 dist-test-slave-qsm9 krb5kdc[8636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:21:13 dist-test-slave-qsm9 krb5kdc[8636](info): set up 2 sockets
Sep 26 20:21:13 dist-test-slave-qsm9 krb5kdc[8636](info): commencing operation
krb5kdc: starting...
W20260926 20:21:14.968652 467 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:21:15 dist-test-slave-qsm9 krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:21:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:21:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:21:15.099996 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:41251
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:41251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:15.212422 8649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:15.212723 8649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:15.212765 8649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:15.216853 8649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:21:15.216913 8649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:15.216933 8649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:15.216950 8649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:21:15.216965 8649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:21:15.222342 8649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:41251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:41251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:15.223553 8649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:15.224598 8649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:15.233417 8657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:15.233654 8649 server_base.cc:1034] running on GCE node
W20260926 20:21:15.233424 8654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:15.233469 8655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:15.234334 8649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:15.235575 8649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:15.236780 8649 hybrid_clock.cc:648] HybridClock initialized: now 1790454075236759 us; error 33 us; skew 500 ppm
Sep 26 20:21:15 dist-test-slave-qsm9 krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:15.245604 8649 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:21:15.246757 8649 webserver.cc:460] Webserver started at http://127.0.116.254:33399/ using document root <none> and password file <none>
I20260926 20:21:15.247371 8649 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:15.247466 8649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:15.247748 8649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:15.249869 8649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "5324f097bcde4409a6ea993d5e3157d4"
format_stamp: "Formatted at 2026-09-26 20:21:15 on dist-test-slave-qsm9"
server_key: "f3b1ff48e7d8d64f81e383e095de2066"
server_key_iv: "97b43c72a26f5da1c83cd707b6c33e7e"
server_key_version: "encryptionkey@0"
I20260926 20:21:15.250437 8649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "5324f097bcde4409a6ea993d5e3157d4"
format_stamp: "Formatted at 2026-09-26 20:21:15 on dist-test-slave-qsm9"
server_key: "f3b1ff48e7d8d64f81e383e095de2066"
server_key_iv: "97b43c72a26f5da1c83cd707b6c33e7e"
server_key_version: "encryptionkey@0"
I20260926 20:21:15.254317 8649 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:21:15.256981 8664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:15.258309 8649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:21:15.258461 8649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "5324f097bcde4409a6ea993d5e3157d4"
format_stamp: "Formatted at 2026-09-26 20:21:15 on dist-test-slave-qsm9"
server_key: "f3b1ff48e7d8d64f81e383e095de2066"
server_key_iv: "97b43c72a26f5da1c83cd707b6c33e7e"
server_key_version: "encryptionkey@0"
I20260926 20:21:15.258585 8649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:15.277554 8649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:15.280997 8649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:15.281220 8649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:15.288674 8649 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:41251
I20260926 20:21:15.288697 8716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:41251 every 8 connection(s)
I20260926 20:21:15.289796 8649 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:21:15.292847 8717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:15.297094 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 8649
I20260926 20:21:15.297253 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:21:15.297549 467 external_mini_cluster.cc:1428] Setting key d99bd562cdf2fc65abc9a9cabff40a4c
I20260926 20:21:15.299118 8717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4: Bootstrap starting.
I20260926 20:21:15.301676 8717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:15.302446 8717 log.cc:824] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4: Log is configured to *not* fsync() on all Append() calls
I20260926 20:21:15.304702 8717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4: No bootstrap required, opened a new log
Sep 26 20:21:15 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:21:15.308919 8717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5324f097bcde4409a6ea993d5e3157d4" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41251 } }
I20260926 20:21:15.309231 8717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:15.309283 8717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5324f097bcde4409a6ea993d5e3157d4, State: Initialized, Role: FOLLOWER
I20260926 20:21:15.309816 8717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5324f097bcde4409a6ea993d5e3157d4" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41251 } }
I20260926 20:21:15.309980 8717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:21:15.310068 8717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:21:15.310168 8717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:15.311305 8717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5324f097bcde4409a6ea993d5e3157d4" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41251 } }
I20260926 20:21:15.311735 8717 leader_election.cc:302] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5324f097bcde4409a6ea993d5e3157d4; no voters:
I20260926 20:21:15.312160 8717 leader_election.cc:288] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:21:15.312245 8722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:21:15.312446 8722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [term 1 LEADER]: Becoming Leader. State: Replica: 5324f097bcde4409a6ea993d5e3157d4, State: Running, Role: LEADER
I20260926 20:21:15.312762 8722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5324f097bcde4409a6ea993d5e3157d4" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41251 } }
I20260926 20:21:15.313519 8720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:15.299053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54096 (local address 127.0.116.254:41251)
0926 20:21:15.299560 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:15.299576 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:15.299612 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:21:15.300248 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:15.300265 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:15.300605 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:15.300902 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:15.301911 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:15.303007 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:15.304000 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:15.304699 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:15.304900 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:15.307600 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:15.307607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:15.307628 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:15.307662 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:15.309751 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:15.310362 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:15.310370 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:15.310381 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:15.310500 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:15.310819 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:15.310823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:15.310824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:15.311589 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:15.311820 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:15.312186 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 20:21:15.312672 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":177,"threads_started":1}
I20260926 20:21:15.313580 8717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:21:15.314020 8722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5324f097bcde4409a6ea993d5e3157d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5324f097bcde4409a6ea993d5e3157d4" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41251 } } }
I20260926 20:21:15.314152 8722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:15.314365 8722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5324f097bcde4409a6ea993d5e3157d4. Latest consensus state: current_term: 1 leader_uuid: "5324f097bcde4409a6ea993d5e3157d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5324f097bcde4409a6ea993d5e3157d4" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41251 } } }
I20260926 20:21:15.314471 8722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:15.314627 8726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:21:15.317811 8726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:21:15.323015 8726 catalog_manager.cc:1347] Generated new cluster ID: ff1f9c6f71124a08a9c1ef6ad8bc84a4
I20260926 20:21:15.323092 8726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:21:15.345065 8726 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:21:15.346113 8726 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:21:15.353212 8726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4: Generated new TSK 0
I20260926 20:21:15.354079 8726 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
I20260926 20:21:15.392879 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:41251
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:15.516110 8743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:15.516429 8743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:15.516503 8743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:15.521052 8743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:15.521200 8743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:15.521310 8743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:21:15.527107 8743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:41251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:15.528369 8743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:15.529469 8743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:21:15.540350 8743 server_base.cc:1034] running on GCE node
W20260926 20:21:15.540321 8748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:15.540307 8751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:15.540314 8749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:15.541033 8743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:15.541725 8743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:15.542927 8743 hybrid_clock.cc:648] HybridClock initialized: now 1790454075542908 us; error 32 us; skew 500 ppm
Sep 26 20:21:15 dist-test-slave-qsm9 krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:15.545765 8743 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:21:15.546638 8743 webserver.cc:460] Webserver started at http://127.0.116.193:37093/ using document root <none> and password file <none>
I20260926 20:21:15.547217 8743 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:15.547307 8743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:15.547598 8743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:15.549628 8743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "eafd2fb3964c4e339246a9dfb2731550"
format_stamp: "Formatted at 2026-09-26 20:21:15 on dist-test-slave-qsm9"
server_key: "c2abd4d23113e927a565865f4c3d8b76"
server_key_iv: "b36fd7fe7c6bd363e03f49567283b0f0"
server_key_version: "encryptionkey@0"
I20260926 20:21:15.550191 8743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "eafd2fb3964c4e339246a9dfb2731550"
format_stamp: "Formatted at 2026-09-26 20:21:15 on dist-test-slave-qsm9"
server_key: "c2abd4d23113e927a565865f4c3d8b76"
server_key_iv: "b36fd7fe7c6bd363e03f49567283b0f0"
server_key_version: "encryptionkey@0"
I20260926 20:21:15.554184 8743 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 20:21:15.557324 8758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:15.559181 8743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:21:15.559316 8743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "eafd2fb3964c4e339246a9dfb2731550"
format_stamp: "Formatted at 2026-09-26 20:21:15 on dist-test-slave-qsm9"
server_key: "c2abd4d23113e927a565865f4c3d8b76"
server_key_iv: "b36fd7fe7c6bd363e03f49567283b0f0"
server_key_version: "encryptionkey@0"
I20260926 20:21:15.559448 8743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:15.573823 8743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:15.577193 8743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:15.577402 8743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:15.578018 8743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:15.578919 8743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:15.579000 8743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:15.579072 8743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:15.579104 8743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:15.588243 8743 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:41597
I20260926 20:21:15.588270 8871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:41597 every 8 connection(s)
I20260926 20:21:15.589157 8743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:21:15.595379 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 8743
I20260926 20:21:15.595549 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:21:15.595835 467 external_mini_cluster.cc:1428] Setting key e881fef81b39c30d8f4fac756617a15c
Sep 26 20:21:15 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:21:15.603507 8720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:15.590974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:32985 (local address 127.0.116.254:41251)
0926 20:21:15.591141 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:15.591144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:15.592005 (+ 861us) server_negotiation.cc:407] Connection header received
0926 20:21:15.593443 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:15.593447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:15.593506 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:15.593619 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:15.594802 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:15.595346 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:15.596650 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:15.596883 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:15.596970 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:15.599886 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:15.599889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:15.599892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:15.599914 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:15.601317 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:15.601843 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:15.601853 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:15.601855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:15.601928 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:15.602349 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:15.602352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:15.602354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:15.602571 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:15.602644 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:15.603186 (+ 542us) server_negotiation.cc:299] Negotiation successful
0926 20:21:15.603315 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:21:15.604344 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:15.591358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.193:32985)
0926 20:21:15.591853 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:15.591897 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:15.593222 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:15.593801 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:15.593809 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:15.594177 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:15.594654 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:15.594667 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:15.595474 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:15.595478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:15.596529 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:15.596543 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:15.597051 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:15.597055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:15.597273 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:15.597917 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:15.597934 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:15.599712 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:15.601438 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:15.601444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:15.601459 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:15.601734 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:15.602023 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:15.602026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:15.602028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:15.602225 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:15.602700 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:15.602705 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:15.602938 (+ 233us) client_negotiation.cc:769] Sending connection context
0926 20:21:15.603122 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 20:21:15.603461 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":180,"threads_started":1}
WARNING: no policy specified for kudu/127.0.116.194@KRBTEST.COM; defaulting to no policy
I20260926 20:21:15.605816 8872 heartbeater.cc:347] Connected to a master server at 127.0.116.254:41251
I20260926 20:21:15.606045 8872 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:15.606570 8872 heartbeater.cc:511] Master 127.0.116.254:41251 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.194@KRBTEST.COM" created.
I20260926 20:21:15.608307 8681 ts_manager.cc:194] Registered new tserver with Master: eafd2fb3964c4e339246a9dfb2731550 (127.0.116.193:41597)
I20260926 20:21:15.609509 8681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:32985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
I20260926 20:21:15.627264 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:41251
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:15.739527 8877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:15.739822 8877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:15.739867 8877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:15.744155 8877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:15.744231 8877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:15.744294 8877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:21:15.749747 8877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:41251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:15.750844 8877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:15.751896 8877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:15.761569 8882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:15.761528 8885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:15.761535 8883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:15.761821 8877 server_base.cc:1034] running on GCE node
I20260926 20:21:15.762305 8877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:15.763026 8877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:15.764215 8877 hybrid_clock.cc:648] HybridClock initialized: now 1790454075764196 us; error 28 us; skew 500 ppm
Sep 26 20:21:15 dist-test-slave-qsm9 krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:15.767418 8877 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:21:15.768435 8877 webserver.cc:460] Webserver started at http://127.0.116.194:37517/ using document root <none> and password file <none>
I20260926 20:21:15.769026 8877 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:15.769119 8877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:15.769352 8877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:15.771442 8877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "8b2a0983e15f453e93cb948babec5d54"
format_stamp: "Formatted at 2026-09-26 20:21:15 on dist-test-slave-qsm9"
server_key: "a5f0c8511debb0a86f47986fbfe323df"
server_key_iv: "96a9c4e2ce9a235cfc02e32858283238"
server_key_version: "encryptionkey@0"
I20260926 20:21:15.772107 8877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "8b2a0983e15f453e93cb948babec5d54"
format_stamp: "Formatted at 2026-09-26 20:21:15 on dist-test-slave-qsm9"
server_key: "a5f0c8511debb0a86f47986fbfe323df"
server_key_iv: "96a9c4e2ce9a235cfc02e32858283238"
server_key_version: "encryptionkey@0"
I20260926 20:21:15.776049 8877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:21:15.778591 8892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:15.779915 8877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:21:15.780085 8877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "8b2a0983e15f453e93cb948babec5d54"
format_stamp: "Formatted at 2026-09-26 20:21:15 on dist-test-slave-qsm9"
server_key: "a5f0c8511debb0a86f47986fbfe323df"
server_key_iv: "96a9c4e2ce9a235cfc02e32858283238"
server_key_version: "encryptionkey@0"
I20260926 20:21:15.780211 8877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:15.800987 8877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:15.804421 8877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:15.804639 8877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:15.805248 8877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:15.806125 8877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:15.806205 8877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:15.806281 8877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:15.806335 8877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:15.815718 8877 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:44955
I20260926 20:21:15.815740 9005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:44955 every 8 connection(s)
I20260926 20:21:15.816669 8877 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:21:15.824287 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 8877
I20260926 20:21:15.824455 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:21:15.824770 467 external_mini_cluster.cc:1428] Setting key 8fdae27b37c19a82456db24595c909f5
Sep 26 20:21:15 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:21:15.831911 8720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:15.818442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:56389 (local address 127.0.116.254:41251)
0926 20:21:15.818611 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:15.818615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:15.819515 (+ 900us) server_negotiation.cc:407] Connection header received
0926 20:21:15.820946 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:15.820950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:15.821006 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:15.821109 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:15.822308 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:15.822850 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:15.824099 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:15.824331 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:15.824444 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:15.827699 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:15.827706 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:15.827713 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:15.827750 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:15.829412 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:15.830084 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:15.830094 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:15.830097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:15.830173 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:15.830619 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:15.830625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:15.830630 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:15.830907 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:15.831036 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:15.831568 (+ 532us) server_negotiation.cc:299] Negotiation successful
0926 20:21:15.831713 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 20:21:15.832706 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:15.818834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.194:56389)
0926 20:21:15.819325 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:15.819369 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:15.820726 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:15.821289 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:15.821297 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:15.821664 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:15.822145 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:15.822159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:15.822977 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:15.822980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:15.823969 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:15.823977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:15.824482 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:15.824485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:15.824659 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:15.825393 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:15.825415 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:15.827432 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:15.829559 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:15.829567 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:15.829586 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:15.829971 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:15.830273 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:15.830276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:15.830278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:15.830478 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:15.831029 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:15.831034 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:15.831281 (+ 247us) client_negotiation.cc:769] Sending connection context
0926 20:21:15.831435 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 20:21:15.831773 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":122,"threads_started":1}
I20260926 20:21:15.834007 9006 heartbeater.cc:347] Connected to a master server at 127.0.116.254:41251
I20260926 20:21:15.834287 9006 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:15.834913 9006 heartbeater.cc:511] Master 127.0.116.254:41251 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.116.195@KRBTEST.COM; defaulting to no policy
I20260926 20:21:15.836248 8681 ts_manager.cc:194] Registered new tserver with Master: 8b2a0983e15f453e93cb948babec5d54 (127.0.116.194:44955)
I20260926 20:21:15.836833 8681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:56389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
I20260926 20:21:15.855162 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:41251
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:15.989216 9011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:15.989665 9011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:15.989779 9011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:15.997965 9011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:15.998054 9011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:15.998121 9011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:21:16.004880 9011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:41251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:16.006446 9011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:16.007953 9011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:16.021260 9017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:16.021260 9016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:16.021476 9011 server_base.cc:1034] running on GCE node
W20260926 20:21:16.021557 9019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:16.022377 9011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:16.023290 9011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:16.024509 9011 hybrid_clock.cc:648] HybridClock initialized: now 1790454076024485 us; error 61 us; skew 500 ppm
Sep 26 20:21:16 dist-test-slave-qsm9 krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:16.028319 9011 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:21:16.029312 9011 webserver.cc:460] Webserver started at http://127.0.116.195:43387/ using document root <none> and password file <none>
I20260926 20:21:16.029914 9011 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:16.030020 9011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:16.030258 9011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:16.032449 9011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "0c01ec099eb6473a8dfaf4156fb0d2ea"
format_stamp: "Formatted at 2026-09-26 20:21:16 on dist-test-slave-qsm9"
server_key: "cc2f249910cdddcd4301bb34ba688cee"
server_key_iv: "c58e61b49cfb39be27eebf8d489b7c27"
server_key_version: "encryptionkey@0"
I20260926 20:21:16.033035 9011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "0c01ec099eb6473a8dfaf4156fb0d2ea"
format_stamp: "Formatted at 2026-09-26 20:21:16 on dist-test-slave-qsm9"
server_key: "cc2f249910cdddcd4301bb34ba688cee"
server_key_iv: "c58e61b49cfb39be27eebf8d489b7c27"
server_key_version: "encryptionkey@0"
I20260926 20:21:16.036906 9011 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:21:16.039441 9026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:16.040836 9011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 20:21:16.041042 9011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "0c01ec099eb6473a8dfaf4156fb0d2ea"
format_stamp: "Formatted at 2026-09-26 20:21:16 on dist-test-slave-qsm9"
server_key: "cc2f249910cdddcd4301bb34ba688cee"
server_key_iv: "c58e61b49cfb39be27eebf8d489b7c27"
server_key_version: "encryptionkey@0"
I20260926 20:21:16.041174 9011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:16.069139 9011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:16.072115 9011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:16.072340 9011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:16.072930 9011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:16.073783 9011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:16.073855 9011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:16.073928 9011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:16.073972 9011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:16.083083 9011 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:35869
I20260926 20:21:16.083103 9139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:35869 every 8 connection(s)
I20260926 20:21:16.084018 9011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:21:16.092832 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 9011
I20260926 20:21:16.093006 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:21:16.093382 467 external_mini_cluster.cc:1428] Setting key e6050eb33ae7f7e7692b911e9042a6c4
Sep 26 20:21:16 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:21:16.097671 8720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:16.085917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:51905 (local address 127.0.116.254:41251)
0926 20:21:16.086063 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:16.086066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:16.086916 (+ 850us) server_negotiation.cc:407] Connection header received
0926 20:21:16.088286 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:16.088288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:16.088337 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:16.088412 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:16.089578 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.090124 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.091254 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.091456 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.091607 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:16.094196 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:16.094199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:16.094202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:16.094232 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:16.095920 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:16.096375 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:16.096378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:16.096379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:16.096452 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:16.096754 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:16.096757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:16.096758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:16.096926 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:16.097001 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:16.097455 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 20:21:16.097551 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:21:16.098521 9142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:16.086307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.195:51905)
0926 20:21:16.086754 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:16.086797 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:16.088079 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:16.088584 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:16.088591 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:16.088948 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:16.089441 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:16.089454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.090248 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.090252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:16.091135 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:16.091154 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.091644 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.091646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:16.091783 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:16.092309 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:16.092324 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:16.094002 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:16.096018 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:16.096023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:16.096034 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:16.096290 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:16.096528 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:16.096530 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:16.096531 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:16.096676 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:16.097011 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:16.097015 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:16.097254 (+ 239us) client_negotiation.cc:769] Sending connection context
0926 20:21:16.097410 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 20:21:16.097679 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":115,"threads_started":1}
I20260926 20:21:16.099709 9140 heartbeater.cc:347] Connected to a master server at 127.0.116.254:41251
I20260926 20:21:16.099912 9140 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:16.100366 9140 heartbeater.cc:511] Master 127.0.116.254:41251 requested a full tablet report, sending...
I20260926 20:21:16.101336 8681 ts_manager.cc:194] Registered new tserver with Master: 0c01ec099eb6473a8dfaf4156fb0d2ea (127.0.116.195:35869)
I20260926 20:21:16.101871 8681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:51905
I20260926 20:21:16.107795 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/oryx_127.0.116.193.keytab.
Entry for principal oryx/127.0.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/oryx_127.0.116.193.keytab.
WARNING: no policy specified for oryx/127.0.116.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.116.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/oryx_127.0.116.194.keytab.
Entry for principal oryx/127.0.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/oryx_127.0.116.194.keytab.
WARNING: no policy specified for oryx/127.0.116.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.116.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/oryx_127.0.116.195.keytab.
Entry for principal oryx/127.0.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/oryx_127.0.116.195.keytab.
I20260926 20:21:16.187448 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 8743
I20260926 20:21:16.193202 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 8877
I20260926 20:21:16.198547 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 9011
I20260926 20:21:16.204016 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 8649
I20260926 20:21:16.210395 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:41251
--webserver_interface=127.0.116.254
--webserver_port=33399
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:41251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:16.322877 9149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:16.323233 9149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:16.323319 9149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:16.327533 9149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:21:16.327633 9149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:16.327668 9149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:16.327705 9149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:21:16.327739 9149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:21:16.333137 9149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:41251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:41251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=33399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:16.334259 9149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:16.335305 9149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:16.343869 9154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:16.343941 9155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:16.344013 9157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:16.344210 9149 server_base.cc:1034] running on GCE node
I20260926 20:21:16.344874 9149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:16.346096 9149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:16.347297 9149 hybrid_clock.cc:648] HybridClock initialized: now 1790454076347282 us; error 28 us; skew 500 ppm
Sep 26 20:21:16 dist-test-slave-qsm9 krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:16.350242 9149 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:21:16.351228 9149 webserver.cc:460] Webserver started at http://127.0.116.254:33399/ using document root <none> and password file <none>
I20260926 20:21:16.351894 9149 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:16.352010 9149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:16.356058 9149 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:21:16.358227 9164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:16.359440 9149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:21:16.359614 9149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "5324f097bcde4409a6ea993d5e3157d4"
format_stamp: "Formatted at 2026-09-26 20:21:15 on dist-test-slave-qsm9"
server_key: "f3b1ff48e7d8d64f81e383e095de2066"
server_key_iv: "97b43c72a26f5da1c83cd707b6c33e7e"
server_key_version: "encryptionkey@0"
I20260926 20:21:16.360190 9149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:16.374213 9149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:16.377461 9149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:16.377699 9149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:16.385210 9149 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:41251
I20260926 20:21:16.385238 9216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:41251 every 8 connection(s)
I20260926 20:21:16.386349 9149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:21:16.386991 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 9149
I20260926 20:21:16.387432 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:41597
--local_ip_for_outbound_sockets=127.0.116.193
--tserver_master_addrs=127.0.116.254:41251
--webserver_port=37093
--webserver_interface=127.0.116.193
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.116.193
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/oryx_127.0.116.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:21:16.390903 9217 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:21:16.392705 9217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4: Bootstrap starting.
I20260926 20:21:16.409317 9217 log.cc:824] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4: Log is configured to *not* fsync() on all Append() calls
I20260926 20:21:16.415957 9217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:21:16.416338 9217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4: Bootstrap complete.
I20260926 20:21:16.420608 9217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5324f097bcde4409a6ea993d5e3157d4" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41251 } }
I20260926 20:21:16.421036 9217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5324f097bcde4409a6ea993d5e3157d4, State: Initialized, Role: FOLLOWER
I20260926 20:21:16.421571 9217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5324f097bcde4409a6ea993d5e3157d4" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41251 } }
I20260926 20:21:16.421715 9217 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:21:16.421799 9217 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:21:16.421911 9217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:21:16.423952 9217 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5324f097bcde4409a6ea993d5e3157d4" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41251 } }
I20260926 20:21:16.424285 9217 leader_election.cc:302] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5324f097bcde4409a6ea993d5e3157d4; no voters:
I20260926 20:21:16.424592 9217 leader_election.cc:288] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:21:16.424789 9221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:21:16.425036 9221 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [term 2 LEADER]: Becoming Leader. State: Replica: 5324f097bcde4409a6ea993d5e3157d4, State: Running, Role: LEADER
I20260926 20:21:16.425352 9221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5324f097bcde4409a6ea993d5e3157d4" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41251 } }
I20260926 20:21:16.425839 9217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:21:16.426842 9222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5324f097bcde4409a6ea993d5e3157d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5324f097bcde4409a6ea993d5e3157d4" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41251 } } }
I20260926 20:21:16.426970 9222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:16.427269 9223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5324f097bcde4409a6ea993d5e3157d4. Latest consensus state: current_term: 2 leader_uuid: "5324f097bcde4409a6ea993d5e3157d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5324f097bcde4409a6ea993d5e3157d4" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41251 } } }
I20260926 20:21:16.427367 9223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:16.430428 9237 catalog_manager.cc:1259] Loaded cluster ID: ff1f9c6f71124a08a9c1ef6ad8bc84a4
I20260926 20:21:16.430505 9237 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4: loading cluster ID for follower catalog manager: success
I20260926 20:21:16.432305 9237 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4: acquiring CA information for follower catalog manager: success
I20260926 20:21:16.433413 9237 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:21:16.433559 9230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:21:16.434433 9230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:21:16.434873 9230 catalog_manager.cc:1259] Loaded cluster ID: ff1f9c6f71124a08a9c1ef6ad8bc84a4
I20260926 20:21:16.434947 9230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:21:16.435575 9230 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:21:16.436045 9230 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5324f097bcde4409a6ea993d5e3157d4: Loaded TSK: 0
I20260926 20:21:16.436698 9230 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:21:16.517025 9219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:16.517310 9219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:16.517355 9219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:16.521559 9219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:16.521637 9219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:16.521737 9219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:21:16.527558 9219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/oryx_127.0.116.193.keytab
--principal=oryx/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:41597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=37093
--tserver_master_addrs=127.0.116.254:41251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:16.528743 9219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:16.529800 9219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:16.540233 9243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:16.540241 9246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:16.540704 9244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:16.540705 9219 server_base.cc:1034] running on GCE node
I20260926 20:21:16.541285 9219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:16.542101 9219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:16.543303 9219 hybrid_clock.cc:648] HybridClock initialized: now 1790454076543262 us; error 51 us; skew 500 ppm
Sep 26 20:21:16 dist-test-slave-qsm9 krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454076, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:16.546695 9219 init.cc:377] Logged in from keytab as oryx/127.0.116.193@KRBTEST.COM (short username oryx)
I20260926 20:21:16.547694 9219 webserver.cc:460] Webserver started at http://127.0.116.193:37093/ using document root <none> and password file <none>
I20260926 20:21:16.548254 9219 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:16.548326 9219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:16.552338 9219 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:21:16.554841 9253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:16.556380 9219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:21:16.556542 9219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "eafd2fb3964c4e339246a9dfb2731550"
format_stamp: "Formatted at 2026-09-26 20:21:15 on dist-test-slave-qsm9"
server_key: "c2abd4d23113e927a565865f4c3d8b76"
server_key_iv: "b36fd7fe7c6bd363e03f49567283b0f0"
server_key_version: "encryptionkey@0"
I20260926 20:21:16.557161 9219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:16.569360 9219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:16.572741 9219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:16.572949 9219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:16.573554 9219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:16.574476 9219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:16.574545 9219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:16.574616 9219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:16.574667 9219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:16.585086 9219 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:41597
I20260926 20:21:16.585119 9366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:41597 every 8 connection(s)
I20260926 20:21:16.586612 9219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:21:16.594468 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 9219
I20260926 20:21:16.595033 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:44955
--local_ip_for_outbound_sockets=127.0.116.194
--tserver_master_addrs=127.0.116.254:41251
--webserver_port=37517
--webserver_interface=127.0.116.194
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.116.194
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/oryx_127.0.116.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:21:16 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:16 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:16.602382 9370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:16.589265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.193:50497)
0926 20:21:16.589759 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:16.589800 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:16.591080 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:16.592305 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:16.592312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:16.592757 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:16.593237 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:16.593252 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.594765 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.594770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:16.596301 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:16.596316 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.597494 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.597497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:16.597640 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:16.598301 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:16.598327 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:16.601270 (+ 2943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":138,"threads_started":1}
W20260926 20:21:16.602375 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:16.589095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:50497 (local address 127.0.116.254:41251)
0926 20:21:16.589535 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:16.589567 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:16.589922 (+ 355us) server_negotiation.cc:407] Connection header received
0926 20:21:16.591552 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:16.591565 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:16.591887 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:16.592226 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:16.593420 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.594584 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.596461 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.597321 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.597521 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:16.601522 (+ 4001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.193:50497: BlockingRecv error: recv got EOF from 127.0.116.193:50497 (error 108)
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":112,"threads_started":1}
W20260926 20:21:16.602988 9367 heartbeater.cc:644] Failed to heartbeat to 127.0.116.254:41251 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.116.254:41251: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Sep 26 20:21:16 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:16 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:16.609047 9370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:16.603555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.193:60765)
0926 20:21:16.603680 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:16.603696 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:16.603790 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:16.604128 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:16.604131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:16.604336 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:16.604495 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:16.604502 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.605266 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.605273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:16.605853 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:16.605861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.606325 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.606329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:16.606429 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:16.606960 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:16.606984 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:16.608912 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 20:21:16.609210 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:16.603623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:60765 (local address 127.0.116.254:41251)
0926 20:21:16.603736 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:16.603741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:16.603756 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:21:16.603905 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:16.603908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:16.603960 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:16.604114 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:16.604609 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.605146 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.605990 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.606214 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.606325 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:16.609058 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.193:60765: BlockingRecv error: recv got EOF from 127.0.116.193:60765 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 20:21:16 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:16 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:16.615162 9370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:16.609631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.193:57203)
0926 20:21:16.609733 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:16.609748 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:16.609822 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:16.610062 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:16.610065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:16.610227 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:16.610316 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:16.610322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.611005 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.611009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:16.611962 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:16.611973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.612441 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.612445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:16.612533 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:16.613101 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:16.613118 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:16.615003 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 20:21:16.615409 9367 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:21:16.615579 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:16.609688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:57203 (local address 127.0.116.254:41251)
0926 20:21:16.609796 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:16.609799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:16.609811 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:21:16.609910 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:16.609912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:16.609950 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:16.610027 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:16.610418 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.610909 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.612104 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.612320 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.612408 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:16.615129 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.193:57203: BlockingRecv error: recv got EOF from 127.0.116.193:57203 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 20:21:16.711997 9371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:16.712316 9371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:16.712394 9371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:16.716498 9371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:16.716611 9371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:16.716717 9371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:21:16.722342 9371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/oryx_127.0.116.194.keytab
--principal=oryx/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:44955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=37517
--tserver_master_addrs=127.0.116.254:41251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:16.723552 9371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:16.724612 9371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:16.735013 9377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:16.735139 9371 server_base.cc:1034] running on GCE node
W20260926 20:21:16.734982 9379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:16.734990 9376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:16.735831 9371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:16.736583 9371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:16.737747 9371 hybrid_clock.cc:648] HybridClock initialized: now 1790454076737732 us; error 28 us; skew 500 ppm
Sep 26 20:21:16 dist-test-slave-qsm9 krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454076, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:16.740563 9371 init.cc:377] Logged in from keytab as oryx/127.0.116.194@KRBTEST.COM (short username oryx)
I20260926 20:21:16.741899 9371 webserver.cc:460] Webserver started at http://127.0.116.194:37517/ using document root <none> and password file <none>
I20260926 20:21:16.742573 9371 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:16.742674 9371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:16.746577 9371 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:21:16.748754 9386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:16.749987 9371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:21:16.750108 9371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "8b2a0983e15f453e93cb948babec5d54"
format_stamp: "Formatted at 2026-09-26 20:21:15 on dist-test-slave-qsm9"
server_key: "a5f0c8511debb0a86f47986fbfe323df"
server_key_iv: "96a9c4e2ce9a235cfc02e32858283238"
server_key_version: "encryptionkey@0"
I20260926 20:21:16.750655 9371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:16.778293 9371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:16.781291 9371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:16.781499 9371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:16.782124 9371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:16.783049 9371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:16.783095 9371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:16.783161 9371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:16.783202 9371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:16.792436 9371 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:44955
I20260926 20:21:16.792459 9499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:44955 every 8 connection(s)
I20260926 20:21:16.793677 9371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:21:16 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:16.803040 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 9371
I20260926 20:21:16.803526 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:35869
--local_ip_for_outbound_sockets=127.0.116.195
--tserver_master_addrs=127.0.116.254:41251
--webserver_port=43387
--webserver_interface=127.0.116.195
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.116.195
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/oryx_127.0.116.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:21:16 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:21:16.804888 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:16.795571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:58181 (local address 127.0.116.254:41251)
0926 20:21:16.795732 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:16.795737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:16.796633 (+ 896us) server_negotiation.cc:407] Connection header received
0926 20:21:16.797926 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:16.797929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:16.797980 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:16.798052 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:16.799198 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.799804 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.800880 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.801082 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.801173 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:16.804704 (+ 3531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.194:58181: BlockingRecv error: recv got EOF from 127.0.116.194:58181 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:21:16.805647 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:16.796027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.194:58181)
0926 20:21:16.796476 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:16.796520 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:16.797735 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:16.798237 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:16.798243 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:16.798591 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:16.799053 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:16.799066 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.799908 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.799911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:16.800780 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:16.800789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.801220 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.801223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:16.801355 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:16.801985 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:16.802002 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:16.804490 (+ 2488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":117,"threads_started":1}
W20260926 20:21:16.806173 9500 heartbeater.cc:644] Failed to heartbeat to 127.0.116.254:41251 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.116.254:41251: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Sep 26 20:21:16 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:16 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:16.812163 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:16.806615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.194:40029)
0926 20:21:16.806724 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:16.806739 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:16.806827 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:16.807077 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:16.807080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:16.807264 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:16.807355 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:16.807379 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.808234 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.808239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:16.808789 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:16.808797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.809284 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.809288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:16.809378 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:16.809945 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:16.809956 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:16.811920 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 20:21:16.812182 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:16.806727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:40029 (local address 127.0.116.254:41251)
0926 20:21:16.806860 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:16.806865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:16.806881 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:21:16.806926 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:16.806929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:16.806977 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:16.807060 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:16.807517 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.808128 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.808908 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.809173 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.809283 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:16.812057 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.194:40029: BlockingRecv error: recv got EOF from 127.0.116.194:40029 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 20:21:16 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:16 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:16.817694 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:16.812771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.194:60695)
0926 20:21:16.812864 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:16.812878 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:16.812977 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:16.813216 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:16.813219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:16.813373 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:16.813440 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:16.813445 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.814176 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.814181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:16.814695 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:16.814702 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.815143 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.815146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:16.815217 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:16.815741 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:16.815751 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:16.817542 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 20:21:16.817907 9500 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:21:16.817785 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:16.812878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:60695 (local address 127.0.116.254:41251)
0926 20:21:16.813013 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:16.813019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:16.813032 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:21:16.813068 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:16.813071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:16.813116 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:16.813203 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:16.813547 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.814054 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.814811 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:16.815035 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:16.815148 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:16.817663 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.194:60695: BlockingRecv error: recv got EOF from 127.0.116.194:60695 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 20:21:16.921084 9503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:16.921337 9503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:16.921412 9503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:16.927153 9503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:16.927244 9503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:16.927348 9503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:21:16.934499 9503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/krb5kdc/oryx_127.0.116.195.keytab
--principal=oryx/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:35869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=43387
--tserver_master_addrs=127.0.116.254:41251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:16.935619 9503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:16.936611 9503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:16.946218 9508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:16.946247 9509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:16.946218 9511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:16.946470 9503 server_base.cc:1034] running on GCE node
I20260926 20:21:16.946980 9503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:16.947774 9503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:16.948968 9503 hybrid_clock.cc:648] HybridClock initialized: now 1790454076948948 us; error 31 us; skew 500 ppm
Sep 26 20:21:16 dist-test-slave-qsm9 krb5kdc[8636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454076, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:16.951728 9503 init.cc:377] Logged in from keytab as oryx/127.0.116.195@KRBTEST.COM (short username oryx)
I20260926 20:21:16.952659 9503 webserver.cc:460] Webserver started at http://127.0.116.195:43387/ using document root <none> and password file <none>
I20260926 20:21:16.953233 9503 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:16.953341 9503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:16.957300 9503 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:21:16.959401 9518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:16.960670 9503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:21:16.960824 9503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "0c01ec099eb6473a8dfaf4156fb0d2ea"
format_stamp: "Formatted at 2026-09-26 20:21:16 on dist-test-slave-qsm9"
server_key: "cc2f249910cdddcd4301bb34ba688cee"
server_key_iv: "c58e61b49cfb39be27eebf8d489b7c27"
server_key_version: "encryptionkey@0"
I20260926 20:21:16.961366 9503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:16.983791 9503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:16.986828 9503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:16.987044 9503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:16.987717 9503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:16.988570 9503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:16.988646 9503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:16.988734 9503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:16.988790 9503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:16.997908 9503 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:35869
I20260926 20:21:16.997927 9631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:35869 every 8 connection(s)
I20260926 20:21:16.999020 9503 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:21:17.001288 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 9503
Sep 26 20:21:17 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:17 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:21:17.011471 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:17.000865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:43407 (local address 127.0.116.254:41251)
0926 20:21:17.001007 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:17.001010 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:17.002072 (+ 1062us) server_negotiation.cc:407] Connection header received
0926 20:21:17.003829 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:17.003831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:17.003870 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:17.003932 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:17.005373 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:17.006008 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:17.007187 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:17.007555 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:17.007650 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:17.011335 (+ 3685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.195:43407: BlockingRecv error: recv got EOF from 127.0.116.195:43407 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":51}
I20260926 20:21:17.012096 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:17.002380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54098 (local address 127.0.116.254:41251)
0926 20:21:17.002671 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:17.002676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:17.002690 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:21:17.002741 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:17.002744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:17.002805 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:17.002920 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:17.003667 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:17.004457 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:17.005593 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:17.005878 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:17.005959 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:17.007675 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:17.007683 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:17.007703 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:17.007738 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:17.009704 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:17.010164 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:17.010175 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:17.010180 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:17.010328 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:17.010609 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:17.010614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:17.010616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:17.011064 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:17.011269 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:17.011615 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 20:21:17.011979 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
I20260926 20:21:17.012404 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:17.001308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.195:43407)
0926 20:21:17.001904 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:17.001970 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:17.003572 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:17.004126 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:17.004135 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:17.004646 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:17.005231 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:17.005247 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:17.006129 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:17.006132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:17.007049 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:17.007058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:17.007700 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:17.007704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:17.007911 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:17.008495 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:17.008517 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:17.011233 (+ 2716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":152,"threads_started":1}
W20260926 20:21:17.012871 9632 heartbeater.cc:644] Failed to heartbeat to 127.0.116.254:41251 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.116.254:41251: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Sep 26 20:21:17 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:17 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:17.019133 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:17.013357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.195:33497)
0926 20:21:17.013476 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:17.013492 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:17.013576 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:17.013897 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:17.013901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:17.014116 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:17.014257 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:17.014264 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:17.015427 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:17.015429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:17.016080 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:17.016090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:17.016506 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:17.016509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:17.016579 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:17.017097 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:17.017109 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:17.018968 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 20:21:17.019282 9636 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:17.013444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:33497 (local address 127.0.116.254:41251)
0926 20:21:17.013542 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:17.013546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:17.013560 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:21:17.013663 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:17.013667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:17.013709 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:17.013789 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:17.014607 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:17.015347 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:17.016194 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:17.016406 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:17.016482 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:17.019164 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.195:33497: BlockingRecv error: recv got EOF from 127.0.116.195:33497 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 26 20:21:17 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:17 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:17.025503 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:17.020042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.195:55405)
0926 20:21:17.020333 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:17.020350 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:17.020429 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:17.020804 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:17.020807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:17.020990 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:17.021072 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:17.021077 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:17.022062 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:17.022065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:17.022697 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:17.022706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:17.023094 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:17.023097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:17.023183 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:17.023704 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:17.023718 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:17.025401 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217}
W20260926 20:21:17.025733 9632 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:21:17.025842 9636 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:17.020180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:55405 (local address 127.0.116.254:41251)
0926 20:21:17.020379 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:17.020384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:17.020401 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:21:17.020544 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:17.020550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:17.020601 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:17.020682 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:17.021166 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:17.021933 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:17.022798 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:17.023000 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:17.023079 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:17.025548 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.195:55405: BlockingRecv error: recv got EOF from 127.0.116.195:55405 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 26 20:21:17 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:17 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:17.621132 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:17.616044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.193:56503)
0926 20:21:17.616312 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:17.616326 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:17.616416 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:17.616640 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:17.616643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:17.616836 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:17.616932 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:17.616937 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:17.617691 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:17.617694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:17.618264 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:17.618271 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:17.618697 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:17.618700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:17.618761 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:17.619248 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:17.619259 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:17.621007 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":90,"threads_started":1}
W20260926 20:21:17.621239 9638 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:17.616122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:56503 (local address 127.0.116.254:41251)
0926 20:21:17.616371 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:17.616374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:17.616387 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:21:17.616485 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:17.616487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:17.616529 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:17.616607 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:17.617032 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:17.617587 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:17.618370 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:17.618591 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:17.618685 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:17.621139 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.193:56503: BlockingRecv error: recv got EOF from 127.0.116.193:56503 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":60,"threads_started":1}
Sep 26 20:21:17 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:17 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:17.824177 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:17.818577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.194:43169)
0926 20:21:17.818871 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:17.818885 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:17.818975 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:17.819261 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:17.819263 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:17.819438 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:17.819613 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:17.819619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:17.820429 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:17.820432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:17.821028 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:17.821079 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:17.821495 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:17.821497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:17.821562 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:17.822104 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:17.822115 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:17.824025 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":124,"threads_started":1}
W20260926 20:21:17.824354 9638 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:17.818739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:43169 (local address 127.0.116.254:41251)
0926 20:21:17.819024 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:17.819028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:17.819044 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:21:17.819084 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:17.819086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:17.819129 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:17.819214 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:17.819728 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:17.820301 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:17.821185 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:17.821396 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:17.821475 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:17.824188 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.194:43169: BlockingRecv error: recv got EOF from 127.0.116.194:43169 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 20:21:18 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:18 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:18.037051 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:18.031257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.195:48203)
0926 20:21:18.031579 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:18.031595 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:18.031754 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:18.032092 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:18.032105 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:18.032336 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:18.032493 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:18.032501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:18.033446 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:18.033448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:18.034014 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:18.034020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:18.034492 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:18.034495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:18.034590 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:18.035108 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:18.035119 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:18.036928 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":107,"threads_started":1}
W20260926 20:21:18.037189 9638 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:18.031324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:48203 (local address 127.0.116.254:41251)
0926 20:21:18.031478 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:18.031484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:18.031666 (+ 182us) server_negotiation.cc:407] Connection header received
0926 20:21:18.031871 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:18.031874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:18.031959 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:18.032054 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:18.032648 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:18.033327 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:18.034147 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:18.034377 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:18.034492 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:18.037069 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.195:48203: BlockingRecv error: recv got EOF from 127.0.116.195:48203 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 20:21:18 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:18 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:18.627343 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:18.622012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.193:55985)
0926 20:21:18.622278 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:18.622300 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:18.622387 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:18.622641 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:18.622643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:18.622836 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:18.622934 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:18.622939 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:18.623672 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:18.623674 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:18.624242 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:18.624250 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:18.624658 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:18.624671 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:18.624729 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:18.625221 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:18.625232 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:18.627205 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":95,"threads_started":1}
W20260926 20:21:18.627523 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:18.622097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:55985 (local address 127.0.116.254:41251)
0926 20:21:18.622352 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:18.622356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:18.622369 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:21:18.622467 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:18.622469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:18.622512 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:18.622580 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:18.623034 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:18.623572 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:18.624347 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:18.624564 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:18.624658 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:18.627328 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.193:55985: BlockingRecv error: recv got EOF from 127.0.116.193:55985 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":105,"threads_started":1}
Sep 26 20:21:18 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:18 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:18.830724 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:18.824982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.194:55083)
0926 20:21:18.825229 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:18.825244 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:18.825394 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:18.825690 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:18.825692 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:18.825868 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:18.825960 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:18.825965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:18.826898 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:18.826901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:18.827471 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:18.827480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:18.828049 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:18.828052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:18.828114 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:18.828630 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:18.828641 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:18.830560 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":81,"threads_started":1}
W20260926 20:21:18.830909 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:18.825171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:55083 (local address 127.0.116.254:41251)
0926 20:21:18.825301 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:18.825305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:18.825319 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:21:18.825516 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:18.825525 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:18.825573 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:18.825644 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:18.826075 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:18.826793 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:18.827708 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:18.827946 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:18.828041 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:18.830736 (+ 2695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.194:55083: BlockingRecv error: recv got EOF from 127.0.116.194:55083 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 20:21:19 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:19 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:19.043845 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:19.038010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.195:40015)
0926 20:21:19.038324 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:19.038340 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:19.038454 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:19.038914 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:19.038917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:19.039181 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:19.039273 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:19.039280 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:19.040122 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:19.040127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:19.040683 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:19.040690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:19.041147 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:19.041151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:19.041235 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:19.041773 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:19.041785 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:19.043657 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":124,"threads_started":1}
W20260926 20:21:19.043886 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:19.038214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:40015 (local address 127.0.116.254:41251)
0926 20:21:19.038335 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:19.038341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:19.038397 (+ 56us) server_negotiation.cc:407] Connection header received
0926 20:21:19.038568 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:19.038571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:19.038625 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:19.038895 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:19.039397 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:19.040014 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:19.040796 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:19.041039 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:19.041146 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:19.043783 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.195:40015: BlockingRecv error: recv got EOF from 127.0.116.195:40015 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 20:21:19 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:19 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:19.634320 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:19.628153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.193:37805)
0926 20:21:19.628538 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:19.628554 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:19.628716 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:19.629009 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:19.629012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:19.629218 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:19.629313 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:19.629319 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:19.630111 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:19.630117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:19.630687 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:19.630696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:19.631152 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:19.631155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:19.631240 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:19.631931 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:19.631943 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:19.634096 (+ 2153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":82,"threads_started":1}
W20260926 20:21:19.634567 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:19.628295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:37805 (local address 127.0.116.254:41251)
0926 20:21:19.628539 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:19.628546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:19.628643 (+ 97us) server_negotiation.cc:407] Connection header received
0926 20:21:19.628825 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:19.628829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:19.628885 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:19.628980 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:19.629435 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:19.629982 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:19.630809 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:19.631037 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:19.631147 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:19.634281 (+ 3134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.193:37805: BlockingRecv error: recv got EOF from 127.0.116.193:37805 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":58,"threads_started":1}
Sep 26 20:21:19 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:19 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:19.837255 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:19.831616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.194:51687)
0926 20:21:19.831870 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:19.831883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:19.831977 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:19.832207 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:19.832209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:19.832390 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:19.832512 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:19.832518 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:19.833407 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:19.833414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:19.834075 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:19.834086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:19.834559 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:19.834562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:19.834630 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:19.835159 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:19.835170 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:19.837129 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":89,"threads_started":1}
W20260926 20:21:19.837493 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:19.831651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:51687 (local address 127.0.116.254:41251)
0926 20:21:19.831788 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:19.831792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:19.831947 (+ 155us) server_negotiation.cc:407] Connection header received
0926 20:21:19.832053 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:19.832056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:19.832100 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:19.832167 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:19.832705 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:19.833242 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:19.834222 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:19.834437 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:19.834519 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:19.837343 (+ 2824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.194:51687: BlockingRecv error: recv got EOF from 127.0.116.194:51687 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 20:21:20 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:20 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:20.050065 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:20.044601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.195:60649)
0926 20:21:20.044877 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:20.044891 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:20.045000 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:20.045240 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:20.045243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:20.045429 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:20.045537 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:20.045542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:20.046271 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:20.046274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:20.046947 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:20.046955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:20.047388 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:20.047391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:20.047457 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:20.048039 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:20.048051 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:20.049919 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":107,"threads_started":1}
W20260926 20:21:20.050141 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:20.044647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:60649 (local address 127.0.116.254:41251)
0926 20:21:20.044788 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:20.044794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:20.044967 (+ 173us) server_negotiation.cc:407] Connection header received
0926 20:21:20.045088 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:20.045090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:20.045137 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:20.045209 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:20.045631 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:20.046140 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:20.047055 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:20.047288 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:20.047366 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:20.050048 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.195:60649: BlockingRecv error: recv got EOF from 127.0.116.195:60649 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 20:21:20 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:20 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:20.644995 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:20.635212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.193:40721)
0926 20:21:20.635579 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:20.635593 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:20.635714 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:20.635976 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:20.635978 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:20.636158 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:20.636269 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:20.636274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:20.637063 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:20.637066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:20.637637 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:20.637646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:20.638138 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:20.638141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:20.638218 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:20.642828 (+ 4610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:20.642843 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:20.644872 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":98,"threads_started":1}
W20260926 20:21:20.645200 9650 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:20.635330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:40721 (local address 127.0.116.254:41251)
0926 20:21:20.635647 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:20.635651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:20.635667 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:21:20.635810 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:20.635812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:20.635856 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:20.635937 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:20.636391 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:20.636950 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:20.637765 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:20.637999 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:20.638082 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:20.645034 (+ 6952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.193:40721: BlockingRecv error: recv got EOF from 127.0.116.193:40721 (error 108)
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":79,"threads_started":1}
Sep 26 20:21:20 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:20 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:20.843767 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:20.838007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.194:53719)
0926 20:21:20.838323 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:20.838340 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:20.838490 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:20.838759 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:20.838762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:20.838970 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:20.839065 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:20.839070 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:20.839869 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:20.839876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:20.840475 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:20.840484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:20.840927 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:20.840930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:20.841023 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:20.841522 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:20.841533 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:20.843534 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":86,"threads_started":1}
W20260926 20:21:20.843863 9650 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:20.838137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:53719 (local address 127.0.116.254:41251)
0926 20:21:20.838314 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:20.838319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:20.838429 (+ 110us) server_negotiation.cc:407] Connection header received
0926 20:21:20.838592 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:20.838596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:20.838651 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:20.838741 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:20.839182 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:20.839769 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:20.840596 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:20.840815 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:20.840922 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:20.843706 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.194:53719: BlockingRecv error: recv got EOF from 127.0.116.194:53719 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 20:21:21 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:21 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:21.056499 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:21.050943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.195:59015)
0926 20:21:21.051252 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:21.051264 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:21.051396 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:21.051698 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:21.051701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:21.051902 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:21.051989 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:21.051995 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:21.052731 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:21.052736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:21.053297 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:21.053304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:21.053786 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:21.053789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:21.053876 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:21.054488 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:21.054507 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:21.056383 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":97,"threads_started":1}
W20260926 20:21:21.056733 9650 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:21.051025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:59015 (local address 127.0.116.254:41251)
0926 20:21:21.051187 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:21.051192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:21.051334 (+ 142us) server_negotiation.cc:407] Connection header received
0926 20:21:21.051531 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:21.051534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:21.051585 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:21.051683 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:21.052099 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:21.052628 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:21.053410 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:21.053650 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:21.053763 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:21.056568 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.195:59015: BlockingRecv error: recv got EOF from 127.0.116.195:59015 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 20:21:21 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:21 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:21.651894 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:21.645930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.193:46467)
0926 20:21:21.646215 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:21.646234 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:21.646359 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:21.646609 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:21.646611 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:21.646826 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:21.646919 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:21.646923 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:21.647718 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:21.647721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:21.648340 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:21.648351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:21.648890 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:21.648892 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:21.648994 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:21.649659 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:21.649674 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:21.651777 (+ 2103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
W20260926 20:21:21.652082 9654 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:21.646122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:46467 (local address 127.0.116.254:41251)
0926 20:21:21.646378 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:21.646382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:21.646399 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:21:21.646439 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:21.646441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:21.646480 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:21.646557 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:21.647029 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:21.647568 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:21.648480 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:21.648779 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:21.648880 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:21.651911 (+ 3031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.193:46467: BlockingRecv error: recv got EOF from 127.0.116.193:46467 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":90,"threads_started":1}
Sep 26 20:21:21 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:21 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:21.850430 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:21.844551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.194:56801)
0926 20:21:21.844831 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:21.844844 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:21.844968 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:21.845254 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:21.845257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:21.845458 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:21.845544 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:21.845549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:21.846323 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:21.846329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:21.846904 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:21.846912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:21.847408 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:21.847412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:21.847534 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:21.848024 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:21.848035 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:21.850294 (+ 2259us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":102,"threads_started":1}
W20260926 20:21:21.850582 9654 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:21.844827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:56801 (local address 127.0.116.254:41251)
0926 20:21:21.844983 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:21.844987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:21.845015 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:21:21.845060 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:21.845062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:21.845106 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:21.845205 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:21.845668 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:21.846213 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:21.847030 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:21.847272 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:21.847390 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:21.850431 (+ 3041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.194:56801: BlockingRecv error: recv got EOF from 127.0.116.194:56801 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 20:21:22 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:22 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:22.063395 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:22.057432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.195:32863)
0926 20:21:22.057776 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:22.057788 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:22.057912 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:22.058206 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:22.058209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:22.058437 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:22.058528 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:22.058534 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:22.059298 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:22.059302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:22.059963 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:22.059972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:22.060481 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:22.060485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:22.060562 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:22.061138 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:22.061150 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:22.063166 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":92,"threads_started":1}
W20260926 20:21:22.063463 9654 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:22.057497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:32863 (local address 127.0.116.254:41251)
0926 20:21:22.057650 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:22.057656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:22.057865 (+ 209us) server_negotiation.cc:407] Connection header received
0926 20:21:22.058030 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:22.058034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:22.058099 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:22.058187 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:22.058653 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:22.059189 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:22.060099 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:22.060370 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:22.060479 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:22.063338 (+ 2859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.195:32863: BlockingRecv error: recv got EOF from 127.0.116.195:32863 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 20:21:22 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:22 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:22.658738 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:22.652782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.193:38325)
0926 20:21:22.653238 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:22.653253 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:22.653353 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:22.653619 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:22.653621 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:22.653869 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:22.653969 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:22.653975 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:22.654756 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:22.654760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:22.655591 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:22.655604 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:22.656056 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:22.656060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:22.656139 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:22.656730 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:22.656745 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:22.658619 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":111,"threads_started":1}
W20260926 20:21:22.658955 9658 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:22.653017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:38325 (local address 127.0.116.254:41251)
0926 20:21:22.653281 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:22.653285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:22.653302 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:21:22.653451 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:22.653453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:22.653497 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:22.653577 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:22.654086 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:22.654626 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:22.655699 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:22.655937 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:22.656110 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:22.658804 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.193:38325: BlockingRecv error: recv got EOF from 127.0.116.193:38325 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":72,"threads_started":1}
Sep 26 20:21:22 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:22 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:22.857647 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:22.851359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.194:55479)
0926 20:21:22.851716 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:22.851734 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:22.851876 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:22.852204 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:22.852208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:22.852452 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:22.852557 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:22.852564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:22.853409 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:22.853416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:22.854046 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:22.854056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:22.854618 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:22.854623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:22.854717 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:22.855353 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:22.855372 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:22.857458 (+ 2086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":136,"threads_started":1}
W20260926 20:21:22.857731 9658 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:22.851559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:55479 (local address 127.0.116.254:41251)
0926 20:21:22.851709 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:22.851714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:22.851831 (+ 117us) server_negotiation.cc:407] Connection header received
0926 20:21:22.852002 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:22.852006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:22.852070 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:22.852183 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:22.852693 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:22.853279 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:22.854184 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:22.854467 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:22.854611 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:22.857608 (+ 2997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.194:55479: BlockingRecv error: recv got EOF from 127.0.116.194:55479 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 20:21:23 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:23 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:23.069896 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:23.064234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.195:39901)
0926 20:21:23.064492 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:23.064506 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:23.064646 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:23.064946 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:23.064948 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:23.065168 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:23.065279 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:23.065285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:23.066061 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:23.066066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:23.066621 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:23.066628 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:23.067098 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:23.067102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:23.067185 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:23.067761 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:23.067773 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:23.069724 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":94,"threads_started":1}
W20260926 20:21:23.069962 9658 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:23.064279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:39901 (local address 127.0.116.254:41251)
0926 20:21:23.064418 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:23.064423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:23.064587 (+ 164us) server_negotiation.cc:407] Connection header received
0926 20:21:23.064746 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:23.064749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:23.064806 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:23.064908 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:23.065404 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:23.065948 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:23.066756 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:23.066983 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:23.067095 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:23.069850 (+ 2755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.195:39901: BlockingRecv error: recv got EOF from 127.0.116.195:39901 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 20:21:23 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:23 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:23.666045 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:23.659707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.193:46027)
0926 20:21:23.660075 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:23.660090 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:23.660199 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:23.660471 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:23.660473 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:23.660662 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:23.660768 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:23.660774 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:23.661618 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:23.661621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:23.662208 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:23.662228 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:23.662725 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:23.662729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:23.662808 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:23.663430 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:23.663451 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:23.665867 (+ 2416us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":135,"threads_started":1}
W20260926 20:21:23.666121 9662 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:23.659782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:46027 (local address 127.0.116.254:41251)
0926 20:21:23.660071 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:23.660076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:23.660171 (+ 95us) server_negotiation.cc:407] Connection header received
0926 20:21:23.660295 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:23.660297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:23.660346 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:23.660432 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:23.660934 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:23.661509 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:23.662333 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:23.662575 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:23.662659 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:23.665994 (+ 3335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.193:46027: BlockingRecv error: recv got EOF from 127.0.116.193:46027 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":121,"threads_started":1}
Sep 26 20:21:23 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:23 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:23.864421 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:23.858435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.194:54915)
0926 20:21:23.858825 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:23.858848 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:23.858951 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:23.859301 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:23.859303 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:23.859486 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:23.859616 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:23.859622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:23.860516 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:23.860524 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:23.861076 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:23.861085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:23.861573 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:23.861576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:23.861664 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:23.862203 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:23.862215 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:23.864252 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":160,"threads_started":1}
W20260926 20:21:23.864547 9662 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:23.858495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:54915 (local address 127.0.116.254:41251)
0926 20:21:23.858640 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:23.858644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:23.858921 (+ 277us) server_negotiation.cc:407] Connection header received
0926 20:21:23.859074 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:23.859085 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:23.859136 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:23.859229 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:23.859741 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:23.860267 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:23.861211 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:23.861448 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:23.861564 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:23.864421 (+ 2857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.194:54915: BlockingRecv error: recv got EOF from 127.0.116.194:54915 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 20:21:24 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:24 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:24.076350 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:24.070800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.195:35671)
0926 20:21:24.071142 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:24.071156 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:24.071255 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:24.071536 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:24.071539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:24.071766 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:24.071856 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:24.071863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:24.072640 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:24.072645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:24.073228 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:24.073236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:24.073692 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:24.073695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:24.073789 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:24.074323 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:24.074335 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:24.076232 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":95,"threads_started":1}
W20260926 20:21:24.076498 9662 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:24.070881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:35671 (local address 127.0.116.254:41251)
0926 20:21:24.071017 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:24.071023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:24.071264 (+ 241us) server_negotiation.cc:407] Connection header received
0926 20:21:24.071318 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:24.071320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:24.071379 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:24.071519 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:24.071985 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:24.072526 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:24.073351 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:24.073580 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:24.073691 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:24.076377 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.195:35671: BlockingRecv error: recv got EOF from 127.0.116.195:35671 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 20:21:24 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:24 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:24.672369 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:24.666970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.193:49385)
0926 20:21:24.667266 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:24.667285 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:24.667395 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:24.667646 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:24.667648 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:24.667848 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:24.667944 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:24.667949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:24.668689 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:24.668691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:24.669260 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:24.669268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:24.669696 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:24.669699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:24.669773 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:24.670360 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:24.670372 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:24.672251 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":85,"threads_started":1}
W20260926 20:21:24.672551 9666 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:24.667136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:49385 (local address 127.0.116.254:41251)
0926 20:21:24.667406 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:24.667410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:24.667443 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:21:24.667477 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:24.667479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:24.667542 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:24.667610 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:24.668047 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:24.668577 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:24.669374 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:24.669597 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:24.669679 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:24.672410 (+ 2731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.193:49385: BlockingRecv error: recv got EOF from 127.0.116.193:49385 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":111,"threads_started":1}
Sep 26 20:21:24 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:24 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:24.871114 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:24.865273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.194:41447)
0926 20:21:24.865662 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:24.865678 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:24.865812 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:24.866091 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:24.866093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:24.866275 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:24.866367 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:24.866372 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:24.867314 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:24.867316 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:24.867894 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:24.867901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:24.868309 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:24.868311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:24.868371 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:24.868942 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:24.868957 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:24.870995 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":102,"threads_started":1}
W20260926 20:21:24.871457 9666 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:24.865335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:41447 (local address 127.0.116.254:41251)
0926 20:21:24.865532 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:24.865536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:24.865796 (+ 260us) server_negotiation.cc:407] Connection header received
0926 20:21:24.865918 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:24.865923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:24.865975 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:24.866052 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:24.866500 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:24.867214 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:24.867988 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:24.868205 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:24.868291 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:24.871215 (+ 2924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.194:41447: BlockingRecv error: recv got EOF from 127.0.116.194:41447 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 20:21:25 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:25 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:25.082971 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:25.077269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.195:56931)
0926 20:21:25.077547 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:25.077561 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:25.077652 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:25.077901 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:25.077903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:25.078076 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:25.078172 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:25.078177 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:25.078960 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:25.078965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:25.079856 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:25.079869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:25.080437 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:25.080439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:25.080504 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:25.081027 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:25.081039 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:25.082850 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":100,"threads_started":1}
W20260926 20:21:25.083122 9666 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:25.077481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:56931 (local address 127.0.116.254:41251)
0926 20:21:25.077622 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:25.077626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:25.077641 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:21:25.077739 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:25.077741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:25.077788 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:25.077865 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:25.078314 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:25.078818 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:25.079987 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:25.080322 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:25.080404 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:25.083001 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.195:56931: BlockingRecv error: recv got EOF from 127.0.116.195:56931 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 20:21:25 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:25 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:25.682721 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:25.673177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.193:53713)
0926 20:21:25.673570 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:25.673587 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:25.673727 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:25.674014 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:25.674017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:25.674250 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:25.674347 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:25.674354 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:25.675133 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:25.675138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:25.675740 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:25.675750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:25.676215 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:25.676226 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:25.676314 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:25.680583 (+ 4269us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:25.680597 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:25.682593 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":101,"threads_started":1}
W20260926 20:21:25.682938 9670 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:25.673240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:53713 (local address 127.0.116.254:41251)
0926 20:21:25.673504 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:25.673508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:25.673672 (+ 164us) server_negotiation.cc:407] Connection header received
0926 20:21:25.673832 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:25.673835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:25.673891 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:25.673987 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:25.674470 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:25.675011 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:25.675864 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:25.676097 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:25.676208 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:25.682750 (+ 6542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.193:53713: BlockingRecv error: recv got EOF from 127.0.116.193:53713 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
Sep 26 20:21:25 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:25 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:25.877750 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:25.871934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.194:45357)
0926 20:21:25.872236 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:25.872252 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:25.872370 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:25.872650 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:25.872653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:25.872872 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:25.872960 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:25.872965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:25.873750 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:25.873757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:25.874297 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:25.874305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:25.874859 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:25.874863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:25.874956 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:25.875651 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:25.875666 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:25.877626 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":108,"threads_started":1}
W20260926 20:21:25.877875 9670 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:25.872086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:45357 (local address 127.0.116.254:41251)
0926 20:21:25.872280 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:25.872287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:25.872306 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:21:25.872478 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:25.872482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:25.872542 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:25.872629 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:25.873075 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:25.873620 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:25.874423 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:25.874643 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:25.874754 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:25.877758 (+ 3004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.194:45357: BlockingRecv error: recv got EOF from 127.0.116.194:45357 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 20:21:26 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:26 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:26.089501 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:26.083796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.195:55055)
0926 20:21:26.084072 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:26.084086 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:26.084180 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:26.084416 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:26.084418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:26.084606 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:26.084698 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:26.084703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:26.085426 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:26.085429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:26.085994 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:26.086001 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:26.086418 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:26.086422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:26.086493 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:26.087078 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:26.087091 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:26.089383 (+ 2292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":111,"threads_started":1}
W20260926 20:21:26.089669 9670 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:26.083905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:55055 (local address 127.0.116.254:41251)
0926 20:21:26.084042 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:26.084046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:26.084152 (+ 106us) server_negotiation.cc:407] Connection header received
0926 20:21:26.084260 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:26.084263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:26.084306 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:26.084377 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:26.084797 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:26.085327 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:26.086093 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:26.086298 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:26.086371 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:26.089508 (+ 3137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.195:55055: BlockingRecv error: recv got EOF from 127.0.116.195:55055 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 20:21:26 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:26 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:26.689246 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:26.683643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.193:40639)
0926 20:21:26.683944 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:26.683958 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:26.684039 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:26.684289 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:26.684291 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:26.684466 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:26.684560 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:26.684565 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:26.685278 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:26.685281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:26.685827 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:26.685835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:26.686231 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:26.686234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:26.686292 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:26.686842 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:26.686857 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:26.689128 (+ 2271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":89,"threads_started":1}
W20260926 20:21:26.689399 9674 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:26.683788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:40639 (local address 127.0.116.254:41251)
0926 20:21:26.684014 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:26.684018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:26.684032 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:21:26.684130 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:26.684132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:26.684176 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:26.684258 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:26.684658 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:26.685187 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:26.685924 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:26.686141 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:26.686213 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:26.689264 (+ 3051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.193:40639: BlockingRecv error: recv got EOF from 127.0.116.193:40639 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":55,"threads_started":1}
Sep 26 20:21:26 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:26 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:26.884330 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:26.878611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.194:41205)
0926 20:21:26.878908 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:26.878921 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:26.879054 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:26.879374 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:26.879377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:26.879620 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:26.879715 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:26.879721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:26.880435 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:26.880438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:26.880990 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:26.880997 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:26.881475 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:26.881477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:26.881537 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:26.882025 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:26.882035 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:26.884202 (+ 2167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":88,"threads_started":1}
W20260926 20:21:26.884565 9674 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:26.878649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:41205 (local address 127.0.116.254:41251)
0926 20:21:26.878814 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:26.878819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:26.879003 (+ 184us) server_negotiation.cc:407] Connection header received
0926 20:21:26.879181 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:26.879184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:26.879240 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:26.879325 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:26.879812 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:26.880337 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:26.881157 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:26.881376 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:26.881462 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:26.884412 (+ 2950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.194:41205: BlockingRecv error: recv got EOF from 127.0.116.194:41205 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 20:21:27 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:27 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:27.095878 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:27.090379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.195:54815)
0926 20:21:27.090666 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:27.090683 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:27.090797 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:27.091078 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:27.091081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:27.091295 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:27.091384 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:27.091389 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:27.092196 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:27.092202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:27.092772 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:27.092781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:27.093266 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:27.093270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:27.093358 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:27.093858 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:27.093870 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:27.095763 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":103,"threads_started":1}
W20260926 20:21:27.096064 9674 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:27.090509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:54815 (local address 127.0.116.254:41251)
0926 20:21:27.090654 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:27.090659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:27.090781 (+ 122us) server_negotiation.cc:407] Connection header received
0926 20:21:27.090887 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:27.090891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:27.090948 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:27.091051 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:27.091524 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:27.092080 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:27.092899 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:27.093140 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:27.093251 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:27.095954 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.195:54815: BlockingRecv error: recv got EOF from 127.0.116.195:54815 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 20:21:27 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:27 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:27.695573 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:27.690237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.193:52311)
0926 20:21:27.690510 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:27.690523 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:27.690609 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:27.690838 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:27.690839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:27.691020 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:27.691111 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:27.691115 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:27.691893 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:27.691896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:27.692446 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:27.692454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:27.692880 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:27.692883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:27.692941 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:27.693466 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:27.693478 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:27.695356 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":99,"threads_started":1}
W20260926 20:21:27.695689 9678 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:27.690317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:52311 (local address 127.0.116.254:41251)
0926 20:21:27.690584 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:27.690587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:27.690603 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:21:27.690699 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:27.690701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:27.690745 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:27.690828 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:27.691221 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:27.691797 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:27.692563 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:27.692782 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:27.692864 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:27.695518 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.193:52311: BlockingRecv error: recv got EOF from 127.0.116.193:52311 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":99,"threads_started":1}
Sep 26 20:21:27 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:27 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:27.890825 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:27.885066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.194:33905)
0926 20:21:27.885385 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:27.885402 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:27.885544 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:27.885827 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:27.885830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:27.886033 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:27.886122 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:27.886127 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:27.886931 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:27.886937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:27.887526 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:27.887536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:27.887992 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:27.887996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:27.888080 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:27.888581 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:27.888592 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:27.890642 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":82,"threads_started":1}
W20260926 20:21:27.890865 9678 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:27.885189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:33905 (local address 127.0.116.254:41251)
0926 20:21:27.885366 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:27.885371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:27.885481 (+ 110us) server_negotiation.cc:407] Connection header received
0926 20:21:27.885643 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:27.885647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:27.885703 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:27.885790 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:27.886267 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:27.886818 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:27.887648 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:27.887874 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:27.887986 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:27.890768 (+ 2782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.194:33905: BlockingRecv error: recv got EOF from 127.0.116.194:33905 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 20:21:28 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:28 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:28.102595 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:28.096951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.195:55839)
0926 20:21:28.097254 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:28.097268 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:28.097399 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:28.097720 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:28.097724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:28.097931 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:28.098019 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:28.098025 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:28.098796 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:28.098800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:28.099351 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:28.099359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:28.099865 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:28.099869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:28.099954 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:28.100443 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:28.100454 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:28.102385 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":89,"threads_started":1}
W20260926 20:21:28.102635 9678 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:28.096977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:55839 (local address 127.0.116.254:41251)
0926 20:21:28.097143 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:28.097148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:28.097352 (+ 204us) server_negotiation.cc:407] Connection header received
0926 20:21:28.097534 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:28.097537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:28.097604 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:28.097695 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:28.098135 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:28.098675 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:28.099468 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:28.099746 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:28.099866 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:28.102522 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.195:55839: BlockingRecv error: recv got EOF from 127.0.116.195:55839 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 20:21:28 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:28 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:28.702212 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:28.696709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.193:48747)
0926 20:21:28.697016 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:28.697030 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:28.697120 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:28.697373 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:28.697375 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:28.697550 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:28.697643 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:28.697648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:28.698476 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:28.698478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:28.699026 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:28.699034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:28.699615 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:28.699619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:28.699700 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:28.700326 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:28.700339 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:28.702096 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":107,"threads_started":1}
W20260926 20:21:28.702437 9682 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:28.696851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:48747 (local address 127.0.116.254:41251)
0926 20:21:28.697093 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:28.697097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:28.697111 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:21:28.697203 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:28.697206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:28.697248 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:28.697332 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:28.697746 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:28.698357 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:28.699112 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:28.699448 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:28.699596 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:28.702242 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.193:48747: BlockingRecv error: recv got EOF from 127.0.116.193:48747 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":90,"threads_started":1}
Sep 26 20:21:28 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:28 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:28.896715 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:28.891713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.194:57311)
0926 20:21:28.891978 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:28.891991 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:28.892077 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:28.892286 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:28.892288 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:28.892462 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:28.892555 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:28.892560 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:28.893254 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:28.893256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:28.893835 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:28.893843 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:28.894249 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:28.894252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:28.894308 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:28.894788 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:28.894799 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:28.896600 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":98,"threads_started":1}
W20260926 20:21:28.896857 9682 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:28.891922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:57311 (local address 127.0.116.254:41251)
0926 20:21:28.892062 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:28.892066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:28.892079 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:21:28.892145 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:28.892147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:28.892188 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:28.892254 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:28.892654 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:28.893164 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:28.893941 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:28.894150 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:28.894224 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:28.896733 (+ 2509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.194:57311: BlockingRecv error: recv got EOF from 127.0.116.194:57311 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 20:21:29 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:29 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:29.109212 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:29.103571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.195:58289)
0926 20:21:29.103860 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:29.103874 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:29.104010 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:29.104307 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:29.104311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:29.104520 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:29.104613 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:29.104619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:29.105406 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:29.105411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:29.105981 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:29.105989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:29.106444 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:29.106447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:29.106539 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:29.107128 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:29.107141 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:29.109039 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":98,"threads_started":1}
W20260926 20:21:29.109259 9682 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:29.103627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:58289 (local address 127.0.116.254:41251)
0926 20:21:29.103806 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:29.103811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:29.103957 (+ 146us) server_negotiation.cc:407] Connection header received
0926 20:21:29.104117 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:29.104121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:29.104176 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:29.104262 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:29.104748 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:29.105287 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:29.106101 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:29.106332 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:29.106442 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:29.109159 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.195:58289: BlockingRecv error: recv got EOF from 127.0.116.195:58289 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 20:21:29 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:29 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:29.709143 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:29.703184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.193:52371)
0926 20:21:29.703476 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:29.703534 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:29.703654 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:29.703921 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:29.703923 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:29.704189 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:29.704331 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:29.704338 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:29.705277 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:29.705281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:29.705899 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:29.705909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:29.706381 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:29.706385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:29.706470 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:29.707064 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:29.707076 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:29.708950 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":98,"threads_started":1}
W20260926 20:21:29.709223 9686 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:29.703278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:52371 (local address 127.0.116.254:41251)
0926 20:21:29.703555 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:29.703559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:29.703600 (+ 41us) server_negotiation.cc:407] Connection header received
0926 20:21:29.703741 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:29.703744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:29.703795 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:29.703892 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:29.704470 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:29.705150 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:29.706022 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:29.706255 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:29.706364 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:29.709086 (+ 2722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.193:52371: BlockingRecv error: recv got EOF from 127.0.116.193:52371 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":95,"threads_started":1}
Sep 26 20:21:29 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:29 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:29.903174 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:29.897550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.194:60495)
0926 20:21:29.897819 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:29.897837 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:29.897934 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:29.898172 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:29.898174 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:29.898371 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:29.898464 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:29.898469 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:29.899234 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:29.899238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:29.899944 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:29.899956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:29.900462 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:29.900466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:29.900534 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:29.901102 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:29.901114 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:29.903044 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":96,"threads_started":1}
W20260926 20:21:29.903404 9686 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:29.897713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:60495 (local address 127.0.116.254:41251)
0926 20:21:29.897857 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:29.897861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:29.897887 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:21:29.898018 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:29.898020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:29.898063 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:29.898132 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:29.898564 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:29.899099 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:29.900066 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:29.900293 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:29.900374 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:29.903259 (+ 2885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.194:60495: BlockingRecv error: recv got EOF from 127.0.116.194:60495 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 20:21:30 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:30 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:30.116051 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:30.110129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.195:41791)
0926 20:21:30.110558 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:30.110579 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:30.110704 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:30.111023 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:30.111026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:30.111241 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:30.111335 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:30.111341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:30.112159 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:30.112164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:30.112727 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:30.112735 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:30.113193 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:30.113196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:30.113299 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:30.113859 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:30.113871 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:30.115870 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":103,"threads_started":1}
W20260926 20:21:30.116113 9686 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:30.110172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:41791 (local address 127.0.116.254:41251)
0926 20:21:30.110327 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:30.110332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:30.110654 (+ 322us) server_negotiation.cc:407] Connection header received
0926 20:21:30.110861 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:30.110864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:30.110913 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:30.111000 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:30.111457 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:30.112044 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:30.112850 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:30.113079 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:30.113189 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:30.115998 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.195:41791: BlockingRecv error: recv got EOF from 127.0.116.195:41791 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 20:21:30 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:30 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:30.720343 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:30.710078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.193:53047)
0926 20:21:30.710404 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:30.710424 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:30.710558 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:30.710858 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:30.710859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:30.711043 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:30.711171 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:30.711177 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:30.712106 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:30.712108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:30.712673 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:30.712681 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:30.713210 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:30.713213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:30.713322 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:30.717999 (+ 4677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:30.718014 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:30.720182 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":72,"threads_started":1}
W20260926 20:21:30.720543 9690 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:30.710242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:53047 (local address 127.0.116.254:41251)
0926 20:21:30.710606 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:30.710610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:30.710650 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:21:30.710693 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:30.710695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:30.710740 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:30.710830 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:30.711289 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:30.711971 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:30.712778 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:30.713017 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:30.713158 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:30.720362 (+ 7204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.193:53047: BlockingRecv error: recv got EOF from 127.0.116.193:53047 (error 108)
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":82,"threads_started":1}
Sep 26 20:21:30 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:30 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:30.909092 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:30.904028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.194:50245)
0926 20:21:30.904310 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:30.904324 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:30.904418 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:30.904626 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:30.904628 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:30.904812 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:30.904905 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:30.904911 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:30.905664 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:30.905667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:30.906217 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:30.906225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:30.906677 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:30.906679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:30.906740 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:30.907209 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:30.907219 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:30.908972 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":101,"threads_started":1}
W20260926 20:21:30.909220 9690 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:30.904099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:50245 (local address 127.0.116.254:41251)
0926 20:21:30.904237 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:30.904241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:30.904372 (+ 131us) server_negotiation.cc:407] Connection header received
0926 20:21:30.904494 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:30.904496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:30.904539 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:30.904595 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:30.905003 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:30.905563 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:30.906344 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:30.906561 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:30.906642 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:30.909102 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.194:50245: BlockingRecv error: recv got EOF from 127.0.116.194:50245 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 20:21:31 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:31 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.195@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:31.122522 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:31.116972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.195:47655)
0926 20:21:31.117397 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:31.117409 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:31.117529 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:31.117849 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:31.117853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:31.118089 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:31.118195 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:31.118200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:31.118946 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:31.118949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:31.119592 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:31.119601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:31.120036 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:31.120038 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:31.120100 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:31.120604 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:31.120616 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:31.122411 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":94,"threads_started":1}
W20260926 20:21:31.122717 9690 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:31.117111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:47655 (local address 127.0.116.254:41251)
0926 20:21:31.117249 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:31.117285 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:31.117499 (+ 214us) server_negotiation.cc:407] Connection header received
0926 20:21:31.117624 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:31.117645 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:31.117706 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:31.117788 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:31.118325 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:31.118850 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:31.119704 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:31.119935 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:31.120019 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:31.122550 (+ 2531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.195:47655: BlockingRecv error: recv got EOF from 127.0.116.195:47655 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 20:21:31 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:31 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.193@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:31.727687 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:31.721466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.193:33209)
0926 20:21:31.721827 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:31.721846 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:31.721993 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:31.722282 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:31.722287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:31.722567 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:31.722704 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:31.722711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:31.723620 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:31.723623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:31.724241 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:31.724250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:31.724711 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:31.724714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:31.724776 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:31.725367 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:31.725422 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:31.727546 (+ 2124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":103,"threads_started":1}
W20260926 20:21:31.727880 9694 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:31.721601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:33209 (local address 127.0.116.254:41251)
0926 20:21:31.721889 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:31.721895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:31.721912 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:21:31.722096 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:31.722099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:31.722143 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:31.722219 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:31.722861 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:31.723427 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:31.724368 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:31.724602 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:31.724688 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:31.727703 (+ 3015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.193:33209: BlockingRecv error: recv got EOF from 127.0.116.193:33209 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":60,"threads_started":1}
Sep 26 20:21:31 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:21:31 dist-test-slave-qsm9 krb5kdc[8636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.116.194@KRBTEST.COM for oryx/127.0.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:21:31.916373 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:31.910190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41251 (local address 127.0.116.194:42553)
0926 20:21:31.910538 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:31.910556 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:31.910675 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:31.911036 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:31.911039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:31.911251 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:31.911342 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:31.911348 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:31.912181 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:31.912187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:31.912756 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:31.912772 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:31.913243 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:31.913247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:31.913323 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:31.913875 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:31.913888 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:31.916172 (+ 2284us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.116.254:41251: Server oryx/127.0.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":107,"threads_started":1}
W20260926 20:21:31.916414 9694 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:31.910366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:42553 (local address 127.0.116.254:41251)
0926 20:21:31.910518 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:31.910525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:31.910802 (+ 277us) server_negotiation.cc:407] Connection header received
0926 20:21:31.910842 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:31.910844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:31.910907 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:31.911016 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:31.911469 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:31.912071 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:31.912894 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:31.913127 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:31.913238 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:31.916314 (+ 3076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.116.194:42553: BlockingRecv error: recv got EOF from 127.0.116.194:42553 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:21:32.004721 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 9219
I20260926 20:21:32.010221 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 9371
I20260926 20:21:32.015846 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 9503
I20260926 20:21:32.020884 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 9149
2026-09-26T20:21:32Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18100 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T20:21:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:21:32Z Disabled control of system clock
I20260926 20:21:32.051167 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:33545
--webserver_interface=127.0.116.254
--webserver_port=0
--builtin_ntp_servers=127.0.116.212:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:33545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 20:21:32.164036 9699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:32.164336 9699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:32.164372 9699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:32.168529 9699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:21:32.168591 9699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:32.168605 9699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:32.168623 9699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:21:32.168638 9699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:21:32.173446 9699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:33545
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:33545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:32.174490 9699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:32.175848 9699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:32.184700 9707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:32.184715 9705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:32.184710 9704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:32.184842 9699 server_base.cc:1034] running on GCE node
I20260926 20:21:32.185555 9699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:32.186717 9699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:32.187930 9699 hybrid_clock.cc:648] HybridClock initialized: now 1790454092187915 us; error 27 us; skew 500 ppm
I20260926 20:21:32.189971 9699 webserver.cc:460] Webserver started at http://127.0.116.254:41223/ using document root <none> and password file <none>
I20260926 20:21:32.190642 9699 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:32.190748 9699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:32.191009 9699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:32.193418 9699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "5d5cdb788ab54de0853dd41f57d540d7"
format_stamp: "Formatted at 2026-09-26 20:21:32 on dist-test-slave-qsm9"
server_key: "575b1a24b7a9067b1e1998d805d6eb0a"
server_key_iv: "d1c886c65b7c0a309ed3264339b26fc9"
server_key_version: "encryptionkey@0"
I20260926 20:21:32.194046 9699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "5d5cdb788ab54de0853dd41f57d540d7"
format_stamp: "Formatted at 2026-09-26 20:21:32 on dist-test-slave-qsm9"
server_key: "575b1a24b7a9067b1e1998d805d6eb0a"
server_key_iv: "d1c886c65b7c0a309ed3264339b26fc9"
server_key_version: "encryptionkey@0"
I20260926 20:21:32.198223 9699 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 20:21:32.200947 9713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:32.202198 9699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:21:32.202374 9699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "5d5cdb788ab54de0853dd41f57d540d7"
format_stamp: "Formatted at 2026-09-26 20:21:32 on dist-test-slave-qsm9"
server_key: "575b1a24b7a9067b1e1998d805d6eb0a"
server_key_iv: "d1c886c65b7c0a309ed3264339b26fc9"
server_key_version: "encryptionkey@0"
I20260926 20:21:32.202503 9699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:32.225811 9699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:32.226581 9699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:32.226825 9699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:32.234051 9699 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:33545
I20260926 20:21:32.234071 9765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:33545 every 8 connection(s)
I20260926 20:21:32.235040 9699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:21:32.237149 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 9699
I20260926 20:21:32.237241 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:21:32.237551 467 external_mini_cluster.cc:1428] Setting key 7d71300e9d832c513433b2f22ffcc120
I20260926 20:21:32.238530 9766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:32.245769 9766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7: Bootstrap starting.
I20260926 20:21:32.248571 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:32.238958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.0.116.254:33545)
0926 20:21:32.239646 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:32.239671 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:32.239717 (+ 46us) server_negotiation.cc:407] Connection header received
0926 20:21:32.240626 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:32.240651 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:32.241229 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:32.241656 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:32.242129 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:32.243602 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:32.244520 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:32.245222 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:32.245388 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:32.246378 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:32.246384 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:32.246401 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:21:32.246916 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:32.246987 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:21:32.247015 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:32.247141 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:32.247293 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 20:21:32.247613 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":110,"threads_started":1}
I20260926 20:21:32.248740 9766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:32.249504 9766 log.cc:824] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7: Log is configured to *not* fsync() on all Append() calls
I20260926 20:21:32.251716 9766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7: No bootstrap required, opened a new log
I20260926 20:21:32.255201 9766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5cdb788ab54de0853dd41f57d540d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33545 } }
I20260926 20:21:32.255450 9766 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:32.255537 9766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d5cdb788ab54de0853dd41f57d540d7, State: Initialized, Role: FOLLOWER
I20260926 20:21:32.256019 9766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5cdb788ab54de0853dd41f57d540d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33545 } }
I20260926 20:21:32.256150 9766 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:21:32.256243 9766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:21:32.256350 9766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:32.257323 9766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5cdb788ab54de0853dd41f57d540d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33545 } }
I20260926 20:21:32.257678 9766 leader_election.cc:302] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d5cdb788ab54de0853dd41f57d540d7; no voters:
I20260926 20:21:32.258009 9766 leader_election.cc:288] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:21:32.258214 9771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:21:32.258539 9771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7 [term 1 LEADER]: Becoming Leader. State: Replica: 5d5cdb788ab54de0853dd41f57d540d7, State: Running, Role: LEADER
I20260926 20:21:32.258854 9771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5cdb788ab54de0853dd41f57d540d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33545 } }
I20260926 20:21:32.259371 9766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:21:32.260668 9773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d5cdb788ab54de0853dd41f57d540d7. Latest consensus state: current_term: 1 leader_uuid: "5d5cdb788ab54de0853dd41f57d540d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5cdb788ab54de0853dd41f57d540d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33545 } } }
I20260926 20:21:32.260797 9773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:32.261157 9772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d5cdb788ab54de0853dd41f57d540d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d5cdb788ab54de0853dd41f57d540d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 33545 } } }
I20260926 20:21:32.261237 9780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:21:32.261282 9772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7 [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:32.264278 9780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:21:32.269788 9780 catalog_manager.cc:1347] Generated new cluster ID: cfeaa181965a4304bcdfc6f107c58e2e
I20260926 20:21:32.269874 9780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:21:32.284467 9780 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:21:32.285473 9780 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:21:32.292634 9780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d5cdb788ab54de0853dd41f57d540d7: Generated new TSK 0
I20260926 20:21:32.293466 9780 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:21:32.301656 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:33545
--builtin_ntp_servers=127.0.116.212:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 20:21:32.417711 9790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:32.417984 9790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:32.418056 9790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:32.422096 9790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:32.422160 9790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:32.422241 9790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:21:32.426967 9790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:33545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:32.428191 9790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:32.429621 9790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:32.439261 9795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:32.439304 9790 server_base.cc:1034] running on GCE node
W20260926 20:21:32.439255 9798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:32.439255 9796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:32.439932 9790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:32.440640 9790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:32.441811 9790 hybrid_clock.cc:648] HybridClock initialized: now 1790454092441783 us; error 49 us; skew 500 ppm
I20260926 20:21:32.443884 9790 webserver.cc:460] Webserver started at http://127.0.116.193:37045/ using document root <none> and password file <none>
I20260926 20:21:32.444494 9790 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:32.444559 9790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:32.444797 9790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:32.446882 9790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "c08e85950bac4736afe04bb7269acbbe"
format_stamp: "Formatted at 2026-09-26 20:21:32 on dist-test-slave-qsm9"
server_key: "fa8689fc14c71d33a81fbca691494141"
server_key_iv: "82f32d57d38ce14be5f35380c8dbe5a5"
server_key_version: "encryptionkey@0"
I20260926 20:21:32.447454 9790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "c08e85950bac4736afe04bb7269acbbe"
format_stamp: "Formatted at 2026-09-26 20:21:32 on dist-test-slave-qsm9"
server_key: "fa8689fc14c71d33a81fbca691494141"
server_key_iv: "82f32d57d38ce14be5f35380c8dbe5a5"
server_key_version: "encryptionkey@0"
I20260926 20:21:32.451167 9790 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:21:32.453850 9804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:32.455200 9790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:21:32.455351 9790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "c08e85950bac4736afe04bb7269acbbe"
format_stamp: "Formatted at 2026-09-26 20:21:32 on dist-test-slave-qsm9"
server_key: "fa8689fc14c71d33a81fbca691494141"
server_key_iv: "82f32d57d38ce14be5f35380c8dbe5a5"
server_key_version: "encryptionkey@0"
I20260926 20:21:32.455476 9790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:32.484568 9790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:32.485328 9790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:32.485538 9790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:32.486156 9790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:32.487145 9790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:32.487216 9790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:32.487298 9790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:32.487352 9790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:32.496208 9790 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:38617
I20260926 20:21:32.496228 9917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:38617 every 8 connection(s)
I20260926 20:21:32.497123 9790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:21:32.498634 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 9790
I20260926 20:21:32.498764 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:21:32.499110 467 external_mini_cluster.cc:1428] Setting key d0aca3d63eed37198235968cbb636b6b
I20260926 20:21:32.501142 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:33545
--builtin_ntp_servers=127.0.116.212:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 20:21:32.508419 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:32.498981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:55089 (local address 127.0.116.254:33545)
0926 20:21:32.499161 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:32.499166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:32.500516 (+ 1350us) server_negotiation.cc:407] Connection header received
0926 20:21:32.501986 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:32.501990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:32.502047 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:32.502127 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:32.503055 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:32.503687 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:32.505570 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:32.506005 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:32.506142 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:32.507730 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:32.507737 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:32.507742 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:21:32.507763 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:32.507792 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:21:32.507806 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:32.507912 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:32.508098 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 20:21:32.508194 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:21:32.509267 9920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:32.499453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33545 (local address 127.0.116.193:55089)
0926 20:21:32.500347 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:32.500401 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:32.501771 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:32.502307 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:32.502314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:32.502401 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:32.502900 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:32.502913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:32.503820 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:32.503826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:32.505374 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:32.505389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:32.506319 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:32.506324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:32.506529 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:32.507386 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:21:32.507388 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:32.507432 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:21:32.507550 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:32.507936 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:32.507945 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:32.507950 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:21:32.508076 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 20:21:32.508336 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":177,"threads_started":1}
I20260926 20:21:32.510708 9918 heartbeater.cc:347] Connected to a master server at 127.0.116.254:33545
I20260926 20:21:32.510998 9918 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:32.511638 9918 heartbeater.cc:511] Master 127.0.116.254:33545 requested a full tablet report, sending...
I20260926 20:21:32.513186 9730 ts_manager.cc:194] Registered new tserver with Master: c08e85950bac4736afe04bb7269acbbe (127.0.116.193:38617)
I20260926 20:21:32.514348 9730 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.116.193:55089
W20260926 20:21:32.617576 9921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:32.617900 9921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:32.617940 9921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:32.622260 9921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:32.622340 9921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:32.622403 9921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:21:32.627385 9921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:33545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:32.628590 9921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:32.630034 9921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:32.640736 9929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:32.640735 9927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:32.640766 9926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:32.641158 9921 server_base.cc:1034] running on GCE node
I20260926 20:21:32.641656 9921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:32.642396 9921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:32.643573 9921 hybrid_clock.cc:648] HybridClock initialized: now 1790454092643542 us; error 44 us; skew 500 ppm
I20260926 20:21:32.645570 9921 webserver.cc:460] Webserver started at http://127.0.116.194:40283/ using document root <none> and password file <none>
I20260926 20:21:32.646194 9921 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:32.646257 9921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:32.646504 9921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:32.648864 9921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "4bc37e2153ee41bc9f9da4062eca4603"
format_stamp: "Formatted at 2026-09-26 20:21:32 on dist-test-slave-qsm9"
server_key: "7c49131a7c20a4bbef3c62df5cab3663"
server_key_iv: "b049dd00e1f476fbf51758f8a2496c5c"
server_key_version: "encryptionkey@0"
I20260926 20:21:32.649456 9921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "4bc37e2153ee41bc9f9da4062eca4603"
format_stamp: "Formatted at 2026-09-26 20:21:32 on dist-test-slave-qsm9"
server_key: "7c49131a7c20a4bbef3c62df5cab3663"
server_key_iv: "b049dd00e1f476fbf51758f8a2496c5c"
server_key_version: "encryptionkey@0"
I20260926 20:21:32.653417 9921 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 20:21:32.656214 9935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:32.657554 9921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:21:32.657719 9921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "4bc37e2153ee41bc9f9da4062eca4603"
format_stamp: "Formatted at 2026-09-26 20:21:32 on dist-test-slave-qsm9"
server_key: "7c49131a7c20a4bbef3c62df5cab3663"
server_key_iv: "b049dd00e1f476fbf51758f8a2496c5c"
server_key_version: "encryptionkey@0"
I20260926 20:21:32.657855 9921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:32.673009 9921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:32.673804 9921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:32.674036 9921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:32.674690 9921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:32.675750 9921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:32.675832 9921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:32.675911 9921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:32.675968 9921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:32.685904 9921 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:40769
I20260926 20:21:32.685925 10048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:40769 every 8 connection(s)
I20260926 20:21:32.686947 9921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:21:32.688971 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 9921
I20260926 20:21:32.689093 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:21:32.689476 467 external_mini_cluster.cc:1428] Setting key 56633930560a8e91c51648f576811c49
I20260926 20:21:32.691721 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:33545
--builtin_ntp_servers=127.0.116.212:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 20:21:32.697943 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:32.689110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:53605 (local address 127.0.116.254:33545)
0926 20:21:32.689254 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:32.689258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:32.690848 (+ 1590us) server_negotiation.cc:407] Connection header received
0926 20:21:32.692469 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:32.692475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:32.692551 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:32.692648 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:32.693905 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:32.694423 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:32.695649 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:32.696063 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:32.696209 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:32.697344 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:32.697352 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:32.697357 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:21:32.697378 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:32.697406 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:21:32.697420 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:32.697513 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:32.697650 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 20:21:32.697729 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:21:32.698844 10051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:32.689760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33545 (local address 127.0.116.194:53605)
0926 20:21:32.690668 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:32.690729 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:32.692251 (+ 1522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:32.692810 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:32.692817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:32.692932 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:32.693732 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:32.693754 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:32.694542 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:32.694545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:32.695473 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:32.695482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:32.696217 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:32.696220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:32.696397 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:32.697063 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:21:32.697065 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:32.697107 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:21:32.697177 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:32.697508 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:32.697515 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:32.697519 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:21:32.697616 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 20:21:32.697854 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":169,"threads_started":1}
I20260926 20:21:32.700130 10049 heartbeater.cc:347] Connected to a master server at 127.0.116.254:33545
I20260926 20:21:32.700413 10049 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:32.701013 10049 heartbeater.cc:511] Master 127.0.116.254:33545 requested a full tablet report, sending...
I20260926 20:21:32.702131 9730 ts_manager.cc:194] Registered new tserver with Master: 4bc37e2153ee41bc9f9da4062eca4603 (127.0.116.194:40769)
I20260926 20:21:32.702715 9730 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.116.194:53605
W20260926 20:21:32.808070 10052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:32.808352 10052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:32.808424 10052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:32.812584 10052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:32.812651 10052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:32.812760 10052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:21:32.817643 10052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:33545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:32.818817 10052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:32.820204 10052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:32.830391 10057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:32.830338 10058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:32.830612 10052 server_base.cc:1034] running on GCE node
W20260926 20:21:32.830719 10060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:32.831139 10052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:32.831887 10052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:32.833086 10052 hybrid_clock.cc:648] HybridClock initialized: now 1790454092833067 us; error 25 us; skew 500 ppm
I20260926 20:21:32.835299 10052 webserver.cc:460] Webserver started at http://127.0.116.195:46123/ using document root <none> and password file <none>
I20260926 20:21:32.836098 10052 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:32.836205 10052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:32.836431 10052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:32.838646 10052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "92e177a02bc246acb7c5831b93f55b9b"
format_stamp: "Formatted at 2026-09-26 20:21:32 on dist-test-slave-qsm9"
server_key: "bc59c326684fa98acb06ced8ce2c2f17"
server_key_iv: "d517b50084ae1f5e570dc68ec888dc43"
server_key_version: "encryptionkey@0"
I20260926 20:21:32.839246 10052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "92e177a02bc246acb7c5831b93f55b9b"
format_stamp: "Formatted at 2026-09-26 20:21:32 on dist-test-slave-qsm9"
server_key: "bc59c326684fa98acb06ced8ce2c2f17"
server_key_iv: "d517b50084ae1f5e570dc68ec888dc43"
server_key_version: "encryptionkey@0"
I20260926 20:21:32.843276 10052 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 20:21:32.846048 10066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:32.847340 10052 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:21:32.847477 10052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "92e177a02bc246acb7c5831b93f55b9b"
format_stamp: "Formatted at 2026-09-26 20:21:32 on dist-test-slave-qsm9"
server_key: "bc59c326684fa98acb06ced8ce2c2f17"
server_key_iv: "d517b50084ae1f5e570dc68ec888dc43"
server_key_version: "encryptionkey@0"
I20260926 20:21:32.847651 10052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:32.863930 10052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:32.864742 10052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:32.864961 10052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:32.865598 10052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:32.866590 10052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:32.866662 10052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:32.866740 10052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:32.866784 10052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:32.876251 10052 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:36171
I20260926 20:21:32.876268 10179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:36171 every 8 connection(s)
I20260926 20:21:32.877255 10052 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:21:32.879068 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 10052
I20260926 20:21:32.879207 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:21:32.879602 467 external_mini_cluster.cc:1428] Setting key 9673e90c426583a0e12ce4f2e406053d
I20260926 20:21:32.887260 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:32.879185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:55157 (local address 127.0.116.254:33545)
0926 20:21:32.879395 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:32.879403 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:32.880672 (+ 1269us) server_negotiation.cc:407] Connection header received
0926 20:21:32.882455 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:32.882458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:32.882514 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:32.882590 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:32.883536 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:32.884067 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:32.885267 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:32.885498 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:32.885609 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:32.886751 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:32.886754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:32.886756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:21:32.886768 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:32.886792 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:21:32.886811 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:32.886888 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:32.887053 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 20:21:32.887117 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:21:32.888187 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:32.879701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:33545 (local address 127.0.116.195:55157)
0926 20:21:32.880533 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:32.880591 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:32.882236 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:32.882780 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:32.882788 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:32.882879 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:32.883377 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:32.883390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:32.884198 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:32.884226 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:32.885134 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:32.885143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:32.885651 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:32.885655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:32.885837 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:32.886468 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:21:32.886469 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:32.886511 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:21:32.886578 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:32.886889 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:32.886900 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:32.886909 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 20:21:32.887043 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 20:21:32.887334 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":114,"threads_started":1}
I20260926 20:21:32.889412 10180 heartbeater.cc:347] Connected to a master server at 127.0.116.254:33545
I20260926 20:21:32.889663 10180 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:32.890213 10180 heartbeater.cc:511] Master 127.0.116.254:33545 requested a full tablet report, sending...
I20260926 20:21:32.891319 9730 ts_manager.cc:194] Registered new tserver with Master: 92e177a02bc246acb7c5831b93f55b9b (127.0.116.195:36171)
I20260926 20:21:32.891940 9730 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.116.195:55157
I20260926 20:21:32.893800 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 20:21:32.896929 9769 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:32.895898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48962 (local address 127.0.116.254:33545)
0926 20:21:32.896033 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:32.896037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:32.896189 (+ 152us) server_negotiation.cc:407] Connection header received
0926 20:21:32.896414 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:32.896417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:32.896459 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:32.896536 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:32.896816 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48962: BlockingRecv error: recv got EOF from 127.0.0.1:48962 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:21:32.897172 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 9790
I20260926 20:21:32.903221 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 9921
I20260926 20:21:32.909076 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 10052
I20260926 20:21:32.914356 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 9699
2026-09-26T20:21:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (892 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T20:21:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:21:32Z Disabled control of system clock
I20260926 20:21:32.944042 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:38911
--webserver_interface=127.0.116.254
--webserver_port=0
--builtin_ntp_servers=127.0.116.212:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:38911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 20:21:33.059257 10191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:33.059561 10191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:33.059600 10191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:33.063618 10191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:21:33.063679 10191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:33.063695 10191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:33.063711 10191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:21:33.063727 10191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:21:33.068588 10191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:38911
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:38911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:33.069796 10191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:33.071141 10191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:33.079826 10197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:33.080034 10191 server_base.cc:1034] running on GCE node
W20260926 20:21:33.079826 10199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:33.079865 10196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:33.080926 10191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:33.082567 10191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:33.083760 10191 hybrid_clock.cc:648] HybridClock initialized: now 1790454093083729 us; error 50 us; skew 500 ppm
I20260926 20:21:33.085896 10191 webserver.cc:460] Webserver started at http://127.0.116.254:42685/ using document root <none> and password file <none>
I20260926 20:21:33.086537 10191 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:33.086640 10191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:33.086912 10191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:33.089144 10191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "43591372129c4cd3816540e483dec75d"
format_stamp: "Formatted at 2026-09-26 20:21:33 on dist-test-slave-qsm9"
server_key: "574327169084b8143826bb89f23f856e"
server_key_iv: "5523636b7e0b6c6c2b0ad64823adbb56"
server_key_version: "encryptionkey@0"
I20260926 20:21:33.089758 10191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "43591372129c4cd3816540e483dec75d"
format_stamp: "Formatted at 2026-09-26 20:21:33 on dist-test-slave-qsm9"
server_key: "574327169084b8143826bb89f23f856e"
server_key_iv: "5523636b7e0b6c6c2b0ad64823adbb56"
server_key_version: "encryptionkey@0"
I20260926 20:21:33.093688 10191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:21:33.096225 10205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:33.097486 10191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:21:33.097633 10191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "43591372129c4cd3816540e483dec75d"
format_stamp: "Formatted at 2026-09-26 20:21:33 on dist-test-slave-qsm9"
server_key: "574327169084b8143826bb89f23f856e"
server_key_iv: "5523636b7e0b6c6c2b0ad64823adbb56"
server_key_version: "encryptionkey@0"
I20260926 20:21:33.097759 10191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:33.102592 10191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:33.103255 10191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:33.103467 10191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:33.110693 10191 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:38911
I20260926 20:21:33.110715 10257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:38911 every 8 connection(s)
I20260926 20:21:33.111795 10191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:21:33.115044 10258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:33.120139 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 10191
I20260926 20:21:33.120275 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:21:33.120546 467 external_mini_cluster.cc:1428] Setting key 7d690d3cbaae923e120c91a3d815af44
I20260926 20:21:33.120779 10258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d: Bootstrap starting.
I20260926 20:21:33.123252 10258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:33.124138 10258 log.cc:824] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:21:33.126163 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:33.121926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44142 (local address 127.0.116.254:38911)
0926 20:21:33.122547 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:33.122564 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:33.122598 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:21:33.123225 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:33.123241 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:33.123718 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:33.124010 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:33.124806 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:33.124813 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:33.124828 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:21:33.124841 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:33.124916 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:21:33.124942 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:33.125060 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:33.125188 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 20:21:33.125479 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":134,"threads_started":1}
I20260926 20:21:33.126327 10258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d: No bootstrap required, opened a new log
I20260926 20:21:33.129904 10258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43591372129c4cd3816540e483dec75d" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 38911 } }
I20260926 20:21:33.130172 10258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:33.130255 10258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43591372129c4cd3816540e483dec75d, State: Initialized, Role: FOLLOWER
I20260926 20:21:33.130708 10258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43591372129c4cd3816540e483dec75d" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 38911 } }
I20260926 20:21:33.130867 10258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:21:33.130954 10258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:21:33.131076 10258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:33.132287 10258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43591372129c4cd3816540e483dec75d" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 38911 } }
I20260926 20:21:33.132675 10258 leader_election.cc:302] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43591372129c4cd3816540e483dec75d; no voters:
I20260926 20:21:33.132958 10258 leader_election.cc:288] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:21:33.133093 10263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:21:33.133337 10263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d [term 1 LEADER]: Becoming Leader. State: Replica: 43591372129c4cd3816540e483dec75d, State: Running, Role: LEADER
I20260926 20:21:33.133668 10263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43591372129c4cd3816540e483dec75d" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 38911 } }
I20260926 20:21:33.134317 10258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:21:33.135385 10265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43591372129c4cd3816540e483dec75d. Latest consensus state: current_term: 1 leader_uuid: "43591372129c4cd3816540e483dec75d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43591372129c4cd3816540e483dec75d" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 38911 } } }
I20260926 20:21:33.135548 10265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:33.135953 10272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:21:33.135914 10264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43591372129c4cd3816540e483dec75d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43591372129c4cd3816540e483dec75d" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 38911 } } }
I20260926 20:21:33.136135 10264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:33.139205 10272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:21:33.145661 10272 catalog_manager.cc:1347] Generated new cluster ID: f4be9cf272de43b1be1a7e67406c1ae1
I20260926 20:21:33.145753 10272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:21:33.176918 10272 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:21:33.177309 10272 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:21:33.189447 10272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43591372129c4cd3816540e483dec75d: Generated new TSK 0
I20260926 20:21:33.190379 10272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:21:33.204869 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:38911
--builtin_ntp_servers=127.0.116.212:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 20:21:33.321060 10282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:33.321412 10282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:33.321458 10282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:33.325553 10282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:33.325618 10282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:33.325677 10282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:21:33.331192 10282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:38911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:33.332439 10282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:33.333932 10282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:33.344611 10290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:33.344594 10287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:33.344597 10288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:33.344816 10282 server_base.cc:1034] running on GCE node
I20260926 20:21:33.345345 10282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:33.346117 10282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:33.347314 10282 hybrid_clock.cc:648] HybridClock initialized: now 1790454093347272 us; error 31 us; skew 500 ppm
I20260926 20:21:33.349444 10282 webserver.cc:460] Webserver started at http://127.0.116.193:38307/ using document root <none> and password file <none>
I20260926 20:21:33.350091 10282 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:33.350165 10282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:33.350423 10282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:33.352793 10282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "5c3840fee5d84cc8adb2b74a57e7bb7b"
format_stamp: "Formatted at 2026-09-26 20:21:33 on dist-test-slave-qsm9"
server_key: "0551b640673be3a887f8cf5fd5566938"
server_key_iv: "ac98ddaec5dabc79e63657d4b52ea8a5"
server_key_version: "encryptionkey@0"
I20260926 20:21:33.353425 10282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "5c3840fee5d84cc8adb2b74a57e7bb7b"
format_stamp: "Formatted at 2026-09-26 20:21:33 on dist-test-slave-qsm9"
server_key: "0551b640673be3a887f8cf5fd5566938"
server_key_iv: "ac98ddaec5dabc79e63657d4b52ea8a5"
server_key_version: "encryptionkey@0"
I20260926 20:21:33.357645 10282 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:21:33.360518 10296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:33.361979 10282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:21:33.362128 10282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "5c3840fee5d84cc8adb2b74a57e7bb7b"
format_stamp: "Formatted at 2026-09-26 20:21:33 on dist-test-slave-qsm9"
server_key: "0551b640673be3a887f8cf5fd5566938"
server_key_iv: "ac98ddaec5dabc79e63657d4b52ea8a5"
server_key_version: "encryptionkey@0"
I20260926 20:21:33.362260 10282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:33.367307 10282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:33.367995 10282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:33.368193 10282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:33.368777 10282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:33.369714 10282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:33.369768 10282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:33.369812 10282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:33.369827 10282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:33.378813 10282 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:43667
I20260926 20:21:33.378834 10409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:43667 every 8 connection(s)
I20260926 20:21:33.379844 10282 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:21:33.381484 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 10282
I20260926 20:21:33.381616 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:21:33.382000 467 external_mini_cluster.cc:1428] Setting key 2f7b9c6a4d11c982add2e575ff7c4312
I20260926 20:21:33.384002 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:38911
--builtin_ntp_servers=127.0.116.212:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 20:21:33.386730 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:33.381807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:56047 (local address 127.0.116.254:38911)
0926 20:21:33.381968 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:33.381974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:33.383252 (+ 1278us) server_negotiation.cc:407] Connection header received
0926 20:21:33.384890 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:33.384896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:33.384945 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:33.385038 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:33.386295 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:33.386299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:33.386304 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:21:33.386322 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:33.386341 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:21:33.386349 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:33.386421 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:33.386550 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 20:21:33.386596 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:21:33.387724 10412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:33.382318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:38911 (local address 127.0.116.193:56047)
0926 20:21:33.383100 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:33.383155 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:33.384641 (+ 1486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:33.385237 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:33.385247 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:33.385372 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:33.386023 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:21:33.386025 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:33.386067 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:21:33.386138 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:33.386410 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:33.386414 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:33.386418 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:21:33.386590 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 20:21:33.386855 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":129,"threads_started":1}
I20260926 20:21:33.388978 10410 heartbeater.cc:347] Connected to a master server at 127.0.116.254:38911
I20260926 20:21:33.389242 10410 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:33.389664 10410 heartbeater.cc:511] Master 127.0.116.254:38911 requested a full tablet report, sending...
I20260926 20:21:33.391398 10221 ts_manager.cc:194] Registered new tserver with Master: 5c3840fee5d84cc8adb2b74a57e7bb7b (127.0.116.193:43667)
W20260926 20:21:33.501849 10413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:33.502203 10413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:33.502286 10413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:33.506362 10413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:33.506435 10413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:33.506492 10413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:21:33.511435 10413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:38911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:33.512698 10413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:33.514205 10413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:33.524526 10418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:33.524659 10413 server_base.cc:1034] running on GCE node
W20260926 20:21:33.524580 10419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:33.524890 10421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:33.525322 10413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:33.526134 10413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:33.527339 10413 hybrid_clock.cc:648] HybridClock initialized: now 1790454093527321 us; error 37 us; skew 500 ppm
I20260926 20:21:33.529318 10413 webserver.cc:460] Webserver started at http://127.0.116.194:34839/ using document root <none> and password file <none>
I20260926 20:21:33.529940 10413 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:33.530035 10413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:33.530301 10413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:33.532512 10413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "2352a80c3f924ccc95c23b18837aae5d"
format_stamp: "Formatted at 2026-09-26 20:21:33 on dist-test-slave-qsm9"
server_key: "a29dbe899e392b09d64d118e7cb0ce19"
server_key_iv: "6b6e040a1f83e9a173b9e6d58cc2bbd0"
server_key_version: "encryptionkey@0"
I20260926 20:21:33.533108 10413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "2352a80c3f924ccc95c23b18837aae5d"
format_stamp: "Formatted at 2026-09-26 20:21:33 on dist-test-slave-qsm9"
server_key: "a29dbe899e392b09d64d118e7cb0ce19"
server_key_iv: "6b6e040a1f83e9a173b9e6d58cc2bbd0"
server_key_version: "encryptionkey@0"
I20260926 20:21:33.537755 10413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:21:33.540580 10427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:33.541980 10413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:21:33.542085 10413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "2352a80c3f924ccc95c23b18837aae5d"
format_stamp: "Formatted at 2026-09-26 20:21:33 on dist-test-slave-qsm9"
server_key: "a29dbe899e392b09d64d118e7cb0ce19"
server_key_iv: "6b6e040a1f83e9a173b9e6d58cc2bbd0"
server_key_version: "encryptionkey@0"
I20260926 20:21:33.542186 10413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:33.546983 10413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:33.547602 10413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:33.547740 10413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:33.548250 10413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:33.549109 10413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:33.549155 10413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:33.549192 10413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:33.549216 10413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:33.558125 10413 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:37903
I20260926 20:21:33.558141 10540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:37903 every 8 connection(s)
I20260926 20:21:33.559082 10413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:21:33.560999 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 10413
I20260926 20:21:33.561131 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:21:33.561458 467 external_mini_cluster.cc:1428] Setting key 88b794a3b4130123fc673ba4569ae433
I20260926 20:21:33.563412 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:38911
--builtin_ntp_servers=127.0.116.212:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 20:21:33.565855 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:33.561061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:58307 (local address 127.0.116.254:38911)
0926 20:21:33.561258 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:33.561262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:33.562479 (+ 1217us) server_negotiation.cc:407] Connection header received
0926 20:21:33.563930 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:33.563933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:33.563975 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:33.564055 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:33.565344 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:33.565347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:33.565349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:21:33.565362 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:33.565383 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:21:33.565392 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:33.565509 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:33.565698 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 20:21:33.565739 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 20:21:33.567099 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:33.561591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:38911 (local address 127.0.116.194:58307)
0926 20:21:33.562317 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:33.562371 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:33.563697 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:33.564234 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:33.564241 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:33.564339 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:33.565001 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:21:33.565004 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:33.565067 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:21:33.565168 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:33.565525 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:33.565538 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:33.565546 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 20:21:33.565743 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 20:21:33.566017 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":123,"threads_started":1}
I20260926 20:21:33.568415 10541 heartbeater.cc:347] Connected to a master server at 127.0.116.254:38911
I20260926 20:21:33.568681 10541 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:33.569231 10541 heartbeater.cc:511] Master 127.0.116.254:38911 requested a full tablet report, sending...
I20260926 20:21:33.570453 10221 ts_manager.cc:194] Registered new tserver with Master: 2352a80c3f924ccc95c23b18837aae5d (127.0.116.194:37903)
W20260926 20:21:33.676067 10544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:33.676375 10544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:33.676450 10544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:33.680541 10544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:33.680621 10544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:33.680716 10544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:21:33.685479 10544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:38911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:33.686580 10544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:33.688031 10544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:33.697990 10549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:33.698100 10544 server_base.cc:1034] running on GCE node
W20260926 20:21:33.697990 10550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:33.698032 10552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:33.698679 10544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:33.699350 10544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:33.700536 10544 hybrid_clock.cc:648] HybridClock initialized: now 1790454093700522 us; error 32 us; skew 500 ppm
I20260926 20:21:33.702654 10544 webserver.cc:460] Webserver started at http://127.0.116.195:39921/ using document root <none> and password file <none>
I20260926 20:21:33.703245 10544 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:33.703308 10544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:33.703475 10544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:33.705586 10544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "8bdbb7511fe84c9fa1ae1bd6a240be5f"
format_stamp: "Formatted at 2026-09-26 20:21:33 on dist-test-slave-qsm9"
server_key: "69579fca35f400795cfdf4ce95cca55b"
server_key_iv: "33e0c1bff563b8a13e7e5111b6d3f5d4"
server_key_version: "encryptionkey@0"
I20260926 20:21:33.706094 10544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "8bdbb7511fe84c9fa1ae1bd6a240be5f"
format_stamp: "Formatted at 2026-09-26 20:21:33 on dist-test-slave-qsm9"
server_key: "69579fca35f400795cfdf4ce95cca55b"
server_key_iv: "33e0c1bff563b8a13e7e5111b6d3f5d4"
server_key_version: "encryptionkey@0"
I20260926 20:21:33.709851 10544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:21:33.712385 10558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:33.713738 10544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:21:33.713840 10544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "8bdbb7511fe84c9fa1ae1bd6a240be5f"
format_stamp: "Formatted at 2026-09-26 20:21:33 on dist-test-slave-qsm9"
server_key: "69579fca35f400795cfdf4ce95cca55b"
server_key_iv: "33e0c1bff563b8a13e7e5111b6d3f5d4"
server_key_version: "encryptionkey@0"
I20260926 20:21:33.713932 10544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:33.718562 10544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:33.719149 10544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:33.719300 10544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:33.719907 10544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:33.720783 10544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:33.720831 10544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:33.720870 10544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:33.720886 10544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:33.729957 10544 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:38683
I20260926 20:21:33.730005 10671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:38683 every 8 connection(s)
I20260926 20:21:33.730929 10544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:21:33.737313 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:33.732875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:42859 (local address 127.0.116.254:38911)
0926 20:21:33.733055 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:33.733060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:33.734228 (+ 1168us) server_negotiation.cc:407] Connection header received
0926 20:21:33.735640 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:33.735643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:33.735691 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:33.735865 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:33.736919 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:33.736922 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:33.736924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:21:33.736936 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:33.736955 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:21:33.736963 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:33.737040 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:33.737153 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 20:21:33.737192 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:21:33.738161 10674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:33.733357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:38911 (local address 127.0.116.195:42859)
0926 20:21:33.734029 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:33.734070 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:33.735362 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:33.736026 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:33.736033 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:33.736128 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:33.736671 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:21:33.736673 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:33.736720 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:21:33.736789 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:33.737036 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:33.737042 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:33.737046 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:21:33.737176 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 20:21:33.737371 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":102,"threads_started":1}
I20260926 20:21:33.739264 10672 heartbeater.cc:347] Connected to a master server at 127.0.116.254:38911
I20260926 20:21:33.739554 10672 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:33.739980 10672 heartbeater.cc:511] Master 127.0.116.254:38911 requested a full tablet report, sending...
I20260926 20:21:33.740304 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 10544
I20260926 20:21:33.740424 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:21:33.740736 467 external_mini_cluster.cc:1428] Setting key 437db5e01fde2a5376d7dee4bfe68f71
I20260926 20:21:33.741034 10221 ts_manager.cc:194] Registered new tserver with Master: 8bdbb7511fe84c9fa1ae1bd6a240be5f (127.0.116.195:38683)
I20260926 20:21:33.742820 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 20:21:33.745745 10261 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:21:33.744861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44154 (local address 127.0.116.254:38911)
0926 20:21:33.744978 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:33.744982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:33.745165 (+ 183us) server_negotiation.cc:407] Connection header received
0926 20:21:33.745299 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:33.745301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:33.745331 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:33.745424 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:33.745602 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44154: BlockingRecv error: recv got EOF from 127.0.0.1:44154 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:21:33.746219 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 10282
I20260926 20:21:33.751631 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 10413
I20260926 20:21:33.757113 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 10544
I20260926 20:21:33.762478 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 10191
2026-09-26T20:21:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (848 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:21:33 dist-test-slave-qsm9 krb5kdc[10682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:21:33 dist-test-slave-qsm9 krb5kdc[10682](info): set up 2 sockets
Sep 26 20:21:33 dist-test-slave-qsm9 krb5kdc[10682](info): commencing operation
krb5kdc: starting...
W20260926 20:21:34.816833 467 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:21:34 dist-test-slave-qsm9 krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:21:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:21:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:21:34.944396 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:41775
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:41775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:35.055156 10695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:35.055452 10695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:35.055562 10695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:35.059579 10695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:21:35.059638 10695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:35.059656 10695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:35.059674 10695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:21:35.059688 10695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:21:35.064955 10695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:41775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:41775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:35.066036 10695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:35.067067 10695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:35.075805 10701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:35.075814 10700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:35.075825 10703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:35.076217 10695 server_base.cc:1034] running on GCE node
I20260926 20:21:35.076890 10695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:35.078125 10695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:35.079327 10695 hybrid_clock.cc:648] HybridClock initialized: now 1790454095079306 us; error 37 us; skew 500 ppm
Sep 26 20:21:35 dist-test-slave-qsm9 krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:35.082381 10695 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:21:35.083269 10695 webserver.cc:460] Webserver started at http://127.0.116.254:35599/ using document root <none> and password file <none>
I20260926 20:21:35.083889 10695 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:35.083971 10695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:35.084200 10695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:35.086395 10695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "55fb37440c7948fd90bc20625585998a"
format_stamp: "Formatted at 2026-09-26 20:21:35 on dist-test-slave-qsm9"
server_key: "7fe0043752a21ed85f097e03b58d6e81"
server_key_iv: "7d201e9a61a4ee340ae01cbf72ad3949"
server_key_version: "encryptionkey@0"
I20260926 20:21:35.087013 10695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "55fb37440c7948fd90bc20625585998a"
format_stamp: "Formatted at 2026-09-26 20:21:35 on dist-test-slave-qsm9"
server_key: "7fe0043752a21ed85f097e03b58d6e81"
server_key_iv: "7d201e9a61a4ee340ae01cbf72ad3949"
server_key_version: "encryptionkey@0"
I20260926 20:21:35.092031 10695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 20:21:35.094660 10710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:35.096077 10695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:21:35.096226 10695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "55fb37440c7948fd90bc20625585998a"
format_stamp: "Formatted at 2026-09-26 20:21:35 on dist-test-slave-qsm9"
server_key: "7fe0043752a21ed85f097e03b58d6e81"
server_key_iv: "7d201e9a61a4ee340ae01cbf72ad3949"
server_key_version: "encryptionkey@0"
I20260926 20:21:35.096357 10695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:35.116371 10695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:35.120169 10695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:35.120404 10695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:35.127961 10695 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:41775
I20260926 20:21:35.128245 10762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:41775 every 8 connection(s)
I20260926 20:21:35.129038 10695 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:21:35.131299 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 10695
I20260926 20:21:35.131417 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:21:35.131748 467 external_mini_cluster.cc:1428] Setting key 55ca2e1d788834f2752354299fa744ab
I20260926 20:21:35.132305 10763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:35.138463 10763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a: Bootstrap starting.
Sep 26 20:21:35 dist-test-slave-qsm9 krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:21:35.140918 10763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:35.141880 10763 log.cc:824] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:21:35.144404 10763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a: No bootstrap required, opened a new log
I20260926 20:21:35.146482 10766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:35.133306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55754 (local address 127.0.116.254:41775)
0926 20:21:35.133753 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:35.133768 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:35.133825 (+ 57us) server_negotiation.cc:407] Connection header received
0926 20:21:35.134489 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:35.134507 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:35.134849 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:35.135194 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:35.136048 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.137056 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.137936 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.138424 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.138608 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:35.141205 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:35.141214 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:35.141235 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:35.141268 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:35.143265 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:35.143773 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:35.143779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:35.143792 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:35.143933 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:35.144211 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:35.144214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:35.144216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:35.144744 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:35.144957 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:35.145194 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 20:21:35.145501 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":145,"threads_started":1}
I20260926 20:21:35.148876 10763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fb37440c7948fd90bc20625585998a" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41775 } }
I20260926 20:21:35.149142 10763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:35.149181 10763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fb37440c7948fd90bc20625585998a, State: Initialized, Role: FOLLOWER
I20260926 20:21:35.149602 10763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fb37440c7948fd90bc20625585998a" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41775 } }
I20260926 20:21:35.149724 10763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:21:35.149770 10763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:21:35.149847 10763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:35.150938 10763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fb37440c7948fd90bc20625585998a" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41775 } }
I20260926 20:21:35.151242 10763 leader_election.cc:302] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55fb37440c7948fd90bc20625585998a; no voters:
I20260926 20:21:35.151577 10763 leader_election.cc:288] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:21:35.151700 10768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:21:35.151942 10768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a [term 1 LEADER]: Becoming Leader. State: Replica: 55fb37440c7948fd90bc20625585998a, State: Running, Role: LEADER
I20260926 20:21:35.152256 10768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fb37440c7948fd90bc20625585998a" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41775 } }
I20260926 20:21:35.152930 10763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:21:35.154091 10770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55fb37440c7948fd90bc20625585998a. Latest consensus state: current_term: 1 leader_uuid: "55fb37440c7948fd90bc20625585998a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fb37440c7948fd90bc20625585998a" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41775 } } }
I20260926 20:21:35.154232 10770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:35.154063 10769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55fb37440c7948fd90bc20625585998a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fb37440c7948fd90bc20625585998a" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41775 } } }
I20260926 20:21:35.154490 10769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a [sys.catalog]: This master's current role is: LEADER
I20260926 20:21:35.154629 10777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:21:35.157789 10777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:21:35.164914 10777 catalog_manager.cc:1347] Generated new cluster ID: 4a12e0e816954e3aab47ed069aceea65
I20260926 20:21:35.165019 10777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:21:35.174695 10777 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:21:35.175768 10777 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:21:35.191466 10777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a: Generated new TSK 0
I20260926 20:21:35.192274 10777 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
I20260926 20:21:35.231148 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:41775
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:35.350279 10789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:35.350560 10789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:35.350606 10789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:35.355027 10789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:35.355098 10789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:35.355201 10789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:21:35.360904 10789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:41775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:35.362174 10789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:35.363274 10789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:35.373418 10795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:35.373418 10797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:35.373421 10794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:35.374253 10789 server_base.cc:1034] running on GCE node
I20260926 20:21:35.374693 10789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:35.375378 10789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:35.376564 10789 hybrid_clock.cc:648] HybridClock initialized: now 1790454095376539 us; error 37 us; skew 500 ppm
Sep 26 20:21:35 dist-test-slave-qsm9 krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:35.379376 10789 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:21:35.380432 10789 webserver.cc:460] Webserver started at http://127.0.116.193:40219/ using document root <none> and password file <none>
I20260926 20:21:35.381016 10789 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:35.381108 10789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:35.381349 10789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:35.383440 10789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "a58f038472fe469494e35adb211a8053"
format_stamp: "Formatted at 2026-09-26 20:21:35 on dist-test-slave-qsm9"
server_key: "a61803920e6df8ebff958ceabf1ff263"
server_key_iv: "ef9f7c89b35d770636e69f8cceea0c68"
server_key_version: "encryptionkey@0"
I20260926 20:21:35.384107 10789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "a58f038472fe469494e35adb211a8053"
format_stamp: "Formatted at 2026-09-26 20:21:35 on dist-test-slave-qsm9"
server_key: "a61803920e6df8ebff958ceabf1ff263"
server_key_iv: "ef9f7c89b35d770636e69f8cceea0c68"
server_key_version: "encryptionkey@0"
I20260926 20:21:35.388294 10789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:21:35.390906 10804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:35.392269 10789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:21:35.392437 10789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "a58f038472fe469494e35adb211a8053"
format_stamp: "Formatted at 2026-09-26 20:21:35 on dist-test-slave-qsm9"
server_key: "a61803920e6df8ebff958ceabf1ff263"
server_key_iv: "ef9f7c89b35d770636e69f8cceea0c68"
server_key_version: "encryptionkey@0"
I20260926 20:21:35.392560 10789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:35.412974 10789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:35.416033 10789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:35.416244 10789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:35.416854 10789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:35.417678 10789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:35.417748 10789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:35.417817 10789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:35.417870 10789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:35.427016 10789 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:44131
I20260926 20:21:35.427052 10917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:44131 every 8 connection(s)
I20260926 20:21:35.427981 10789 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:21:35.428620 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 10789
I20260926 20:21:35.428738 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:21:35.429049 467 external_mini_cluster.cc:1428] Setting key 8c3229b82447d2c1d5bfa6c09535d849
Sep 26 20:21:35 dist-test-slave-qsm9 krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.116.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.194@KRBTEST.COM" created.
I20260926 20:21:35.442513 10766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:35.429956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:44615 (local address 127.0.116.254:41775)
0926 20:21:35.430092 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:35.430096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:35.431321 (+ 1225us) server_negotiation.cc:407] Connection header received
0926 20:21:35.432267 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:35.432271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:35.432333 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:35.432440 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:35.433612 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.434139 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.435280 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.435572 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.435693 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:35.438401 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:35.438404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:35.438407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:35.438431 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:35.439824 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:35.440341 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:35.440347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:35.440349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:35.440431 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:35.440847 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:35.440850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:35.440852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:35.441303 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:35.441530 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:35.442149 (+ 619us) server_negotiation.cc:299] Negotiation successful
0926 20:21:35.442304 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:21:35.443456 10920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:35.430432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41775 (local address 127.0.116.193:44615)
0926 20:21:35.431153 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:35.431193 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:35.432066 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:35.432624 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:35.432632 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:35.432995 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:35.433467 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:35.433480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.434262 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.434266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:35.435154 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:35.435163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.435742 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.435747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:35.435925 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:35.436545 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:35.436562 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:35.438211 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:35.439943 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:35.439949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:35.439966 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:35.440228 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:35.440527 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:35.440532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:35.440534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:35.440735 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:35.441499 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:35.441507 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:35.441873 (+ 366us) client_negotiation.cc:769] Sending connection context
0926 20:21:35.442098 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 20:21:35.442501 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":158,"threads_started":1}
I20260926 20:21:35.445020 10918 heartbeater.cc:347] Connected to a master server at 127.0.116.254:41775
I20260926 20:21:35.445317 10918 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:35.446002 10918 heartbeater.cc:511] Master 127.0.116.254:41775 requested a full tablet report, sending...
I20260926 20:21:35.447924 10727 ts_manager.cc:194] Registered new tserver with Master: a58f038472fe469494e35adb211a8053 (127.0.116.193:44131)
I20260926 20:21:35.449127 10727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:44615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
I20260926 20:21:35.459211 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:41775
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:35.573828 10923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:35.574095 10923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:35.574139 10923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:35.578687 10923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:35.578758 10923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:35.578850 10923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:21:35.584416 10923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:41775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:35.585527 10923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:35.586580 10923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:35.596220 10928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:35.596259 10929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:35.596221 10931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:35.596460 10923 server_base.cc:1034] running on GCE node
I20260926 20:21:35.596936 10923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:35.597637 10923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:35.598815 10923 hybrid_clock.cc:648] HybridClock initialized: now 1790454095598797 us; error 37 us; skew 500 ppm
Sep 26 20:21:35 dist-test-slave-qsm9 krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:35.602005 10923 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:21:35.602882 10923 webserver.cc:460] Webserver started at http://127.0.116.194:41623/ using document root <none> and password file <none>
I20260926 20:21:35.603443 10923 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:35.603560 10923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:35.603811 10923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:35.605895 10923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "8a17b52b9c4b428ba71cc1ec07657543"
format_stamp: "Formatted at 2026-09-26 20:21:35 on dist-test-slave-qsm9"
server_key: "a3d4ba5967b3aaf557ed914384f6f911"
server_key_iv: "15d088c1e385cd46f88d0d4120eab333"
server_key_version: "encryptionkey@0"
I20260926 20:21:35.606441 10923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "8a17b52b9c4b428ba71cc1ec07657543"
format_stamp: "Formatted at 2026-09-26 20:21:35 on dist-test-slave-qsm9"
server_key: "a3d4ba5967b3aaf557ed914384f6f911"
server_key_iv: "15d088c1e385cd46f88d0d4120eab333"
server_key_version: "encryptionkey@0"
I20260926 20:21:35.610226 10923 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260926 20:21:35.612670 10938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:35.613839 10923 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:21:35.613969 10923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "8a17b52b9c4b428ba71cc1ec07657543"
format_stamp: "Formatted at 2026-09-26 20:21:35 on dist-test-slave-qsm9"
server_key: "a3d4ba5967b3aaf557ed914384f6f911"
server_key_iv: "15d088c1e385cd46f88d0d4120eab333"
server_key_version: "encryptionkey@0"
I20260926 20:21:35.614082 10923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:35.624756 10923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:35.627602 10923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:35.627784 10923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:35.628348 10923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:35.629122 10923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:35.629191 10923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:35.629261 10923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:35.629312 10923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:35.638011 10923 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:36645
I20260926 20:21:35.638036 11051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:36645 every 8 connection(s)
I20260926 20:21:35.638988 10923 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:21:35.646003 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 10923
I20260926 20:21:35.646090 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:21:35.646373 467 external_mini_cluster.cc:1428] Setting key 89fe90734d9980df7dc7bb69aedcd33b
Sep 26 20:21:35 dist-test-slave-qsm9 krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:21:35.653000 10766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:35.640852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:59945 (local address 127.0.116.254:41775)
0926 20:21:35.641000 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:35.641004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:35.642012 (+ 1008us) server_negotiation.cc:407] Connection header received
0926 20:21:35.642909 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:35.642914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:35.642983 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:35.643058 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:35.644251 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.644774 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.646244 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.646524 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.646638 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:35.649348 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:35.649353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:35.649354 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:35.649378 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:35.650831 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:35.651373 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:35.651377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:35.651379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:35.651461 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:35.651883 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:35.651888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:35.651890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:35.652083 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:35.652193 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:35.652708 (+ 515us) server_negotiation.cc:299] Negotiation successful
0926 20:21:35.652834 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:21:35.654327 11054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:35.641227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41775 (local address 127.0.116.194:59945)
0926 20:21:35.641882 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:35.641924 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:35.642679 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:35.643230 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:35.643237 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:35.643625 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:35.644125 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:35.644138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.644918 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.644920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:35.646035 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:35.646045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.646693 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.646697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:35.646901 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:35.647591 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:35.647607 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:35.649158 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:35.650965 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:35.650971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:35.650988 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:35.651257 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:35.651620 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:35.651623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:35.651625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:35.651775 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:35.652193 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:35.652201 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:35.652464 (+ 263us) client_negotiation.cc:769] Sending connection context
0926 20:21:35.652665 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 20:21:35.653040 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.0.116.195@KRBTEST.COM; defaulting to no policy
I20260926 20:21:35.655665 11052 heartbeater.cc:347] Connected to a master server at 127.0.116.254:41775
I20260926 20:21:35.655951 11052 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:35.656490 11052 heartbeater.cc:511] Master 127.0.116.254:41775 requested a full tablet report, sending...
I20260926 20:21:35.657637 10727 ts_manager.cc:194] Registered new tserver with Master: 8a17b52b9c4b428ba71cc1ec07657543 (127.0.116.194:36645)
I20260926 20:21:35.658258 10727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:59945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
I20260926 20:21:35.674435 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:41775
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:21:35.789196 11057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:21:35.789477 11057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:21:35.789523 11057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:21:35.793627 11057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:21:35.793686 11057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:21:35.793746 11057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:21:35.799099 11057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:41775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:21:35.800307 11057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:21:35.801412 11057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:21:35.811094 11062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:21:35.811102 11065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:35.811450 11057 server_base.cc:1034] running on GCE node
W20260926 20:21:35.811102 11063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:21:35.811905 11057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:21:35.812601 11057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:21:35.813796 11057 hybrid_clock.cc:648] HybridClock initialized: now 1790454095813783 us; error 45 us; skew 500 ppm
Sep 26 20:21:35 dist-test-slave-qsm9 krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:21:35.820510 11057 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:21:35.821521 11057 webserver.cc:460] Webserver started at http://127.0.116.195:39419/ using document root <none> and password file <none>
I20260926 20:21:35.822110 11057 fs_manager.cc:359] Metadata directory not provided
I20260926 20:21:35.822201 11057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:21:35.822436 11057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:21:35.824647 11057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "0cc3a0e3ce974c8e823968b8b7433259"
format_stamp: "Formatted at 2026-09-26 20:21:35 on dist-test-slave-qsm9"
server_key: "21618bbfda52f4d7d429438dabca1beb"
server_key_iv: "eedd984f36ac413529615574ed275a78"
server_key_version: "encryptionkey@0"
I20260926 20:21:35.825237 11057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "0cc3a0e3ce974c8e823968b8b7433259"
format_stamp: "Formatted at 2026-09-26 20:21:35 on dist-test-slave-qsm9"
server_key: "21618bbfda52f4d7d429438dabca1beb"
server_key_iv: "eedd984f36ac413529615574ed275a78"
server_key_version: "encryptionkey@0"
I20260926 20:21:35.829183 11057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:21:35.831779 11072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:35.833086 11057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:21:35.833231 11057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "0cc3a0e3ce974c8e823968b8b7433259"
format_stamp: "Formatted at 2026-09-26 20:21:35 on dist-test-slave-qsm9"
server_key: "21618bbfda52f4d7d429438dabca1beb"
server_key_iv: "eedd984f36ac413529615574ed275a78"
server_key_version: "encryptionkey@0"
I20260926 20:21:35.833359 11057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:21:35.869062 11057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:21:35.872144 11057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:21:35.872361 11057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:21:35.873067 11057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:21:35.873917 11057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:21:35.873987 11057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:35.874061 11057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:21:35.874112 11057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:21:35.883734 11057 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:40417
I20260926 20:21:35.883767 11185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:40417 every 8 connection(s)
I20260926 20:21:35.884673 11057 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:21:35.891769 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 11057
I20260926 20:21:35.891924 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:21:35.892236 467 external_mini_cluster.cc:1428] Setting key 0b4ba195f078defdfe0369a781e031c1
Sep 26 20:21:35 dist-test-slave-qsm9 krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:21:35.898375 10766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:35.886570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:60651 (local address 127.0.116.254:41775)
0926 20:21:35.886709 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:35.886713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:35.887908 (+ 1195us) server_negotiation.cc:407] Connection header received
0926 20:21:35.888877 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:35.888881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:35.888942 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:35.889050 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:35.890266 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.890826 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.892016 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.892246 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.892364 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:35.895058 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:35.895061 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:35.895064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:35.895088 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:35.896405 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:35.896886 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:35.896889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:35.896890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:35.896966 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:35.897256 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:35.897258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:35.897260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:35.897447 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:35.897525 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:35.898131 (+ 606us) server_negotiation.cc:299] Negotiation successful
0926 20:21:35.898243 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:21:35.899387 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:35.886980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:41775 (local address 127.0.116.195:60651)
0926 20:21:35.887743 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:35.887818 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:35.888649 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:35.889232 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:35.889239 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:35.889616 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:21:35.890102 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:35.890116 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.890955 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.890959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:35.891873 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:35.891882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.892407 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.892411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:35.892587 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:35.893229 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:21:35.893245 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:21:35.894905 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:21:35.896510 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:35.896515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:35.896528 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:35.896786 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:35.897043 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:21:35.897045 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:21:35.897046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:21:35.897179 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:21:35.897534 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:21:35.897539 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:21:35.897881 (+ 342us) client_negotiation.cc:769] Sending connection context
0926 20:21:35.898073 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 20:21:35.898350 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":112,"threads_started":1}
I20260926 20:21:35.900668 11186 heartbeater.cc:347] Connected to a master server at 127.0.116.254:41775
I20260926 20:21:35.900904 11186 heartbeater.cc:464] Registering TS with master...
I20260926 20:21:35.901419 11186 heartbeater.cc:511] Master 127.0.116.254:41775 requested a full tablet report, sending...
I20260926 20:21:35.902421 10727 ts_manager.cc:194] Registered new tserver with Master: 0cc3a0e3ce974c8e823968b8b7433259 (127.0.116.195:40417)
I20260926 20:21:35.902973 10727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:60651
I20260926 20:21:35.906261 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:21:35.915455 10766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:35.908249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55770 (local address 127.0.116.254:41775)
0926 20:21:35.908390 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:35.908394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:35.908576 (+ 182us) server_negotiation.cc:407] Connection header received
0926 20:21:35.908667 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:35.908669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:35.908708 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:35.908775 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:21:35.909480 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.909981 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.910813 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.911021 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.911101 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:35.912527 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:21:35.912530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:21:35.912532 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:21:35.912556 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:21:35.914016 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:35.914414 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:35.914416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:35.914417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:35.914490 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:21:35.914718 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:21:35.914720 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:21:35.914721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:21:35.914932 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:21:35.915031 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:35.915234 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 20:21:35.915334 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 20:21:35.918709 10727 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:21:35.921837 10727 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:21:35.936484 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:35.930392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:44131 (local address 127.0.0.1:58276)
0926 20:21:35.931051 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:35.931104 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:35.931400 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:35.932148 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:35.932151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:35.932184 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:35.932341 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:35.932349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.933836 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.933839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:35.934994 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:35.935000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.935922 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.935925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:35.936028 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:35.936064 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 20:21:35.936180 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 20:21:35.936310 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":21,"spinlock_wait_cycles":428032,"thread_start_us":83,"threads_started":1}
I20260926 20:21:35.936686 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:35.930532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58276 (local address 127.0.116.193:44131)
0926 20:21:35.931366 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:35.931381 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:35.931405 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:21:35.931594 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:35.931601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:35.931818 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:35.931975 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:35.932469 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.933694 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.935102 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.935795 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.935920 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:35.936083 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:35.936378 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 20:21:35.936488 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":119,"threads_started":1}
I20260926 20:21:35.936534 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:35.930743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:40417 (local address 127.0.0.1:56538)
0926 20:21:35.931238 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:35.931250 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:35.931322 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:35.932144 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:35.932150 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:35.932188 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:35.932374 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:35.932380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.933508 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.933513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:35.934977 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:35.934988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.936189 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.936192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:35.936268 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:35.936288 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 20:21:35.936365 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:21:35.936422 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":39,"spinlock_wait_cycles":3840,"thread_start_us":100,"threads_started":1}
I20260926 20:21:35.936869 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:35.931213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56538 (local address 127.0.116.195:40417)
0926 20:21:35.931675 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:35.931688 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:35.931712 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:21:35.931786 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:35.931790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:35.931972 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:35.932527 (+ 555us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:35.932678 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.933378 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.935141 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.936014 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.936161 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:35.936286 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:35.936633 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 20:21:35.936735 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":131,"threads_started":1}
I20260926 20:21:35.937201 11200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:35.930532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:36645 (local address 127.0.0.1:58906)
0926 20:21:35.931087 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:35.931103 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:35.931242 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:35.932378 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:35.932381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:35.932396 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:35.932501 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:35.932506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.933887 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.933889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:35.935072 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:35.935085 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.936339 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.936343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:35.936429 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:35.936451 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:21:35.936921 (+ 470us) client_negotiation.cc:240] Negotiation successful
0926 20:21:35.937060 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":133,"thread_start_us":116,"threads_started":1}
I20260926 20:21:35.937362 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:35.931545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58906 (local address 127.0.116.194:36645)
0926 20:21:35.931968 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:35.931979 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:35.931999 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:21:35.932071 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:35.932075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:35.932237 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:35.932387 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:35.932614 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.933706 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.935195 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:35.936014 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:35.936134 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:35.936272 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:35.936611 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 20:21:35.936731 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":114,"threads_started":1}
I20260926 20:21:35.939978 11120 tablet_service.cc:1467] Processing CreateTablet for tablet 964dc474a5414a49a24d58e639c7ef5f (DEFAULT_TABLE table=test-table [id=61a7194b5f194a0f92db1973f8f4432c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:21:35.940202 10852 tablet_service.cc:1467] Processing CreateTablet for tablet 964dc474a5414a49a24d58e639c7ef5f (DEFAULT_TABLE table=test-table [id=61a7194b5f194a0f92db1973f8f4432c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:21:35.940356 10986 tablet_service.cc:1467] Processing CreateTablet for tablet 964dc474a5414a49a24d58e639c7ef5f (DEFAULT_TABLE table=test-table [id=61a7194b5f194a0f92db1973f8f4432c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:21:35.941048 11120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 964dc474a5414a49a24d58e639c7ef5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:35.941227 10852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 964dc474a5414a49a24d58e639c7ef5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:35.941226 10986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 964dc474a5414a49a24d58e639c7ef5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:21:35.947635 11203 tablet_bootstrap.cc:492] T 964dc474a5414a49a24d58e639c7ef5f P 0cc3a0e3ce974c8e823968b8b7433259: Bootstrap starting.
I20260926 20:21:35.949261 11204 tablet_bootstrap.cc:492] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053: Bootstrap starting.
I20260926 20:21:35.950699 11203 tablet_bootstrap.cc:654] T 964dc474a5414a49a24d58e639c7ef5f P 0cc3a0e3ce974c8e823968b8b7433259: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:35.951575 11204 tablet_bootstrap.cc:654] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:35.951624 11203 log.cc:824] T 964dc474a5414a49a24d58e639c7ef5f P 0cc3a0e3ce974c8e823968b8b7433259: Log is configured to *not* fsync() on all Append() calls
I20260926 20:21:35.951870 11205 tablet_bootstrap.cc:492] T 964dc474a5414a49a24d58e639c7ef5f P 8a17b52b9c4b428ba71cc1ec07657543: Bootstrap starting.
I20260926 20:21:35.952395 11204 log.cc:824] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053: Log is configured to *not* fsync() on all Append() calls
I20260926 20:21:35.954246 11203 tablet_bootstrap.cc:492] T 964dc474a5414a49a24d58e639c7ef5f P 0cc3a0e3ce974c8e823968b8b7433259: No bootstrap required, opened a new log
I20260926 20:21:35.954492 11205 tablet_bootstrap.cc:654] T 964dc474a5414a49a24d58e639c7ef5f P 8a17b52b9c4b428ba71cc1ec07657543: Neither blocks nor log segments found. Creating new log.
I20260926 20:21:35.954504 11203 ts_tablet_manager.cc:1397] T 964dc474a5414a49a24d58e639c7ef5f P 0cc3a0e3ce974c8e823968b8b7433259: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 20:21:35.954653 11204 tablet_bootstrap.cc:492] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053: No bootstrap required, opened a new log
I20260926 20:21:35.954855 11204 ts_tablet_manager.cc:1397] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:21:35.955332 11205 log.cc:824] T 964dc474a5414a49a24d58e639c7ef5f P 8a17b52b9c4b428ba71cc1ec07657543: Log is configured to *not* fsync() on all Append() calls
I20260926 20:21:35.957468 11205 tablet_bootstrap.cc:492] T 964dc474a5414a49a24d58e639c7ef5f P 8a17b52b9c4b428ba71cc1ec07657543: No bootstrap required, opened a new log
I20260926 20:21:35.957662 11205 ts_tablet_manager.cc:1397] T 964dc474a5414a49a24d58e639c7ef5f P 8a17b52b9c4b428ba71cc1ec07657543: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:21:35.957962 11203 raft_consensus.cc:348] T 964dc474a5414a49a24d58e639c7ef5f P 0cc3a0e3ce974c8e823968b8b7433259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58f038472fe469494e35adb211a8053" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44131 } } peers { permanent_uuid: "8a17b52b9c4b428ba71cc1ec07657543" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 36645 } } peers { permanent_uuid: "0cc3a0e3ce974c8e823968b8b7433259" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 40417 } }
I20260926 20:21:35.958236 11203 raft_consensus.cc:374] T 964dc474a5414a49a24d58e639c7ef5f P 0cc3a0e3ce974c8e823968b8b7433259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:35.958292 11203 raft_consensus.cc:729] T 964dc474a5414a49a24d58e639c7ef5f P 0cc3a0e3ce974c8e823968b8b7433259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cc3a0e3ce974c8e823968b8b7433259, State: Initialized, Role: FOLLOWER
I20260926 20:21:35.958765 11203 consensus_queue.cc:260] T 964dc474a5414a49a24d58e639c7ef5f P 0cc3a0e3ce974c8e823968b8b7433259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58f038472fe469494e35adb211a8053" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44131 } } peers { permanent_uuid: "8a17b52b9c4b428ba71cc1ec07657543" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 36645 } } peers { permanent_uuid: "0cc3a0e3ce974c8e823968b8b7433259" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 40417 } }
I20260926 20:21:35.959412 11186 heartbeater.cc:503] Master 127.0.116.254:41775 was elected leader, sending a full tablet report...
I20260926 20:21:35.959860 11203 ts_tablet_manager.cc:1428] T 964dc474a5414a49a24d58e639c7ef5f P 0cc3a0e3ce974c8e823968b8b7433259: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 20:21:35.960414 11204 raft_consensus.cc:348] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58f038472fe469494e35adb211a8053" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44131 } } peers { permanent_uuid: "8a17b52b9c4b428ba71cc1ec07657543" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 36645 } } peers { permanent_uuid: "0cc3a0e3ce974c8e823968b8b7433259" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 40417 } }
I20260926 20:21:35.960909 11204 raft_consensus.cc:374] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:35.960978 11204 raft_consensus.cc:729] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58f038472fe469494e35adb211a8053, State: Initialized, Role: FOLLOWER
I20260926 20:21:35.961447 11204 consensus_queue.cc:260] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58f038472fe469494e35adb211a8053" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44131 } } peers { permanent_uuid: "8a17b52b9c4b428ba71cc1ec07657543" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 36645 } } peers { permanent_uuid: "0cc3a0e3ce974c8e823968b8b7433259" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 40417 } }
I20260926 20:21:35.961614 11205 raft_consensus.cc:348] T 964dc474a5414a49a24d58e639c7ef5f P 8a17b52b9c4b428ba71cc1ec07657543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58f038472fe469494e35adb211a8053" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44131 } } peers { permanent_uuid: "8a17b52b9c4b428ba71cc1ec07657543" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 36645 } } peers { permanent_uuid: "0cc3a0e3ce974c8e823968b8b7433259" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 40417 } }
I20260926 20:21:35.961977 11205 raft_consensus.cc:374] T 964dc474a5414a49a24d58e639c7ef5f P 8a17b52b9c4b428ba71cc1ec07657543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:21:35.962078 11205 raft_consensus.cc:729] T 964dc474a5414a49a24d58e639c7ef5f P 8a17b52b9c4b428ba71cc1ec07657543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a17b52b9c4b428ba71cc1ec07657543, State: Initialized, Role: FOLLOWER
I20260926 20:21:35.962096 10918 heartbeater.cc:503] Master 127.0.116.254:41775 was elected leader, sending a full tablet report...
I20260926 20:21:35.962505 11204 ts_tablet_manager.cc:1428] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260926 20:21:35.962594 11205 consensus_queue.cc:260] T 964dc474a5414a49a24d58e639c7ef5f P 8a17b52b9c4b428ba71cc1ec07657543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58f038472fe469494e35adb211a8053" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44131 } } peers { permanent_uuid: "8a17b52b9c4b428ba71cc1ec07657543" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 36645 } } peers { permanent_uuid: "0cc3a0e3ce974c8e823968b8b7433259" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 40417 } }
I20260926 20:21:35.963302 11052 heartbeater.cc:503] Master 127.0.116.254:41775 was elected leader, sending a full tablet report...
I20260926 20:21:35.963662 11205 ts_tablet_manager.cc:1428] T 964dc474a5414a49a24d58e639c7ef5f P 8a17b52b9c4b428ba71cc1ec07657543: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 20:21:36.024598 11210 raft_consensus.cc:482] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:21:36.024856 11210 raft_consensus.cc:504] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58f038472fe469494e35adb211a8053" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44131 } } peers { permanent_uuid: "8a17b52b9c4b428ba71cc1ec07657543" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 36645 } } peers { permanent_uuid: "0cc3a0e3ce974c8e823968b8b7433259" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 40417 } }
I20260926 20:21:36.026046 11210 leader_election.cc:288] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a17b52b9c4b428ba71cc1ec07657543 (127.0.116.194:36645), 0cc3a0e3ce974c8e823968b8b7433259 (127.0.116.195:40417)
I20260926 20:21:36.030539 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:36.026485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:43779 (local address 127.0.116.195:40417)
0926 20:21:36.026726 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:36.026755 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:36.026969 (+ 214us) server_negotiation.cc:407] Connection header received
0926 20:21:36.027064 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:36.027067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:36.027125 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:36.027229 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:36.027569 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:36.028277 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:36.029454 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:36.029875 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:36.030009 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:36.030065 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:36.030302 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 20:21:36.030398 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260926 20:21:36.030560 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:36.026489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:49497 (local address 127.0.116.194:36645)
0926 20:21:36.026652 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:36.026657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:36.026788 (+ 131us) server_negotiation.cc:407] Connection header received
0926 20:21:36.027056 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:36.027059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:36.027138 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:36.027192 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:21:36.027644 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:36.028259 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:36.029464 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:36.029879 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:36.030014 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:36.030066 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:36.030304 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 20:21:36.030393 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:21:36.030588 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:36.026346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:40417 (local address 127.0.116.193:43779)
0926 20:21:36.026864 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:36.026877 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:36.026966 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:36.027244 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:36.027247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:36.027265 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:36.027415 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:36.027423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:36.028428 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:36.028430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:36.029326 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:36.029335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:36.030003 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:36.030007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:36.030156 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:36.030194 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 20:21:36.030285 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 20:21:36.030422 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":280,"thread_start_us":80,"threads_started":1}
I20260926 20:21:36.030619 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:36.026346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:36645 (local address 127.0.116.193:49497)
0926 20:21:36.026687 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:36.026705 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:21:36.026910 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:21:36.027340 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:21:36.027342 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:21:36.027362 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:21:36.027474 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:36.027501 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:36.028374 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:36.028380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:36.029326 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:21:36.029335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:21:36.030018 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:21:36.030022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:21:36.030156 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:36.030193 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 20:21:36.030284 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 20:21:36.030411 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":98,"thread_start_us":120,"threads_started":1}
I20260926 20:21:36.031356 11006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964dc474a5414a49a24d58e639c7ef5f" candidate_uuid: "a58f038472fe469494e35adb211a8053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a17b52b9c4b428ba71cc1ec07657543" is_pre_election: true
I20260926 20:21:36.031693 11140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964dc474a5414a49a24d58e639c7ef5f" candidate_uuid: "a58f038472fe469494e35adb211a8053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cc3a0e3ce974c8e823968b8b7433259" is_pre_election: true
I20260926 20:21:36.031762 11006 raft_consensus.cc:2454] T 964dc474a5414a49a24d58e639c7ef5f P 8a17b52b9c4b428ba71cc1ec07657543 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a58f038472fe469494e35adb211a8053 in term 0.
I20260926 20:21:36.031955 11140 raft_consensus.cc:2454] T 964dc474a5414a49a24d58e639c7ef5f P 0cc3a0e3ce974c8e823968b8b7433259 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a58f038472fe469494e35adb211a8053 in term 0.
I20260926 20:21:36.032325 10805 leader_election.cc:302] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0cc3a0e3ce974c8e823968b8b7433259, 8a17b52b9c4b428ba71cc1ec07657543, a58f038472fe469494e35adb211a8053; no voters:
I20260926 20:21:36.032641 11210 raft_consensus.cc:2781] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:21:36.032721 11210 raft_consensus.cc:482] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:21:36.032752 11210 raft_consensus.cc:3037] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:36.033847 11210 raft_consensus.cc:504] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58f038472fe469494e35adb211a8053" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44131 } } peers { permanent_uuid: "8a17b52b9c4b428ba71cc1ec07657543" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 36645 } } peers { permanent_uuid: "0cc3a0e3ce974c8e823968b8b7433259" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 40417 } }
I20260926 20:21:36.034235 11210 leader_election.cc:288] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053 [CANDIDATE]: Term 1 election: Requested vote from peers 8a17b52b9c4b428ba71cc1ec07657543 (127.0.116.194:36645), 0cc3a0e3ce974c8e823968b8b7433259 (127.0.116.195:40417)
I20260926 20:21:36.034598 11140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964dc474a5414a49a24d58e639c7ef5f" candidate_uuid: "a58f038472fe469494e35adb211a8053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cc3a0e3ce974c8e823968b8b7433259"
I20260926 20:21:36.034739 11006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964dc474a5414a49a24d58e639c7ef5f" candidate_uuid: "a58f038472fe469494e35adb211a8053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a17b52b9c4b428ba71cc1ec07657543"
I20260926 20:21:36.034742 11140 raft_consensus.cc:3037] T 964dc474a5414a49a24d58e639c7ef5f P 0cc3a0e3ce974c8e823968b8b7433259 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:36.034871 11006 raft_consensus.cc:3037] T 964dc474a5414a49a24d58e639c7ef5f P 8a17b52b9c4b428ba71cc1ec07657543 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:21:36.036123 11140 raft_consensus.cc:2454] T 964dc474a5414a49a24d58e639c7ef5f P 0cc3a0e3ce974c8e823968b8b7433259 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a58f038472fe469494e35adb211a8053 in term 1.
I20260926 20:21:36.036123 11006 raft_consensus.cc:2454] T 964dc474a5414a49a24d58e639c7ef5f P 8a17b52b9c4b428ba71cc1ec07657543 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a58f038472fe469494e35adb211a8053 in term 1.
I20260926 20:21:36.036430 10808 leader_election.cc:302] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cc3a0e3ce974c8e823968b8b7433259, a58f038472fe469494e35adb211a8053; no voters:
I20260926 20:21:36.036633 11210 raft_consensus.cc:2781] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:21:36.036904 11210 raft_consensus.cc:686] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053 [term 1 LEADER]: Becoming Leader. State: Replica: a58f038472fe469494e35adb211a8053, State: Running, Role: LEADER
I20260926 20:21:36.037236 11210 consensus_queue.cc:237] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58f038472fe469494e35adb211a8053" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44131 } } peers { permanent_uuid: "8a17b52b9c4b428ba71cc1ec07657543" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 36645 } } peers { permanent_uuid: "0cc3a0e3ce974c8e823968b8b7433259" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 40417 } }
I20260926 20:21:36.040540 10726 catalog_manager.cc:5495] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053 reported cstate change: term changed from 0 to 1, leader changed from <none> to a58f038472fe469494e35adb211a8053 (127.0.116.193). New cstate: current_term: 1 leader_uuid: "a58f038472fe469494e35adb211a8053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58f038472fe469494e35adb211a8053" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 44131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a17b52b9c4b428ba71cc1ec07657543" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 36645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cc3a0e3ce974c8e823968b8b7433259" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 40417 } health_report { overall_health: UNKNOWN } } }
I20260926 20:21:36.062772 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:36.057890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58280 (local address 127.0.116.193:44131)
0926 20:21:36.058318 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0926 20:21:36.058321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:21:36.058331 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:21:36.058410 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:21:36.058412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:21:36.058490 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:21:36.058610 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:21:36.058890 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:36.059706 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:36.060924 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:21:36.061250 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:21:36.061366 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:21:36.061769 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:21:36.062207 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:21:36.062475 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 20:21:36.062502 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 20:21:36.062589 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355}
I20260926 20:21:36.069875 11140 raft_consensus.cc:1261] T 964dc474a5414a49a24d58e639c7ef5f P 0cc3a0e3ce974c8e823968b8b7433259 [term 1 FOLLOWER]: Refusing update from remote peer a58f038472fe469494e35adb211a8053: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:21:36.070693 11214 consensus_queue.cc:1035] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cc3a0e3ce974c8e823968b8b7433259" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 40417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:21:36.070873 11006 raft_consensus.cc:1261] T 964dc474a5414a49a24d58e639c7ef5f P 8a17b52b9c4b428ba71cc1ec07657543 [term 1 FOLLOWER]: Refusing update from remote peer a58f038472fe469494e35adb211a8053: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:21:36.071533 11214 consensus_queue.cc:1035] T 964dc474a5414a49a24d58e639c7ef5f P a58f038472fe469494e35adb211a8053 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a17b52b9c4b428ba71cc1ec07657543" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 36645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:21:36.080601 11218 mvcc.cc:205] Tried to move back new op lower bound from 7333699977493590016 to 7333699977371615232. Current Snapshot: MvccSnapshot[applied={T|T < 7333699977493590016}]
I20260926 20:21:36.084836 11220 mvcc.cc:205] Tried to move back new op lower bound from 7333699977493590016 to 7333699977371615232. Current Snapshot: MvccSnapshot[applied={T|T < 7333699977493590016}]
I20260926 20:21:36.087646 10727 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:21:36.087821 10727 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:21:36.090595 10727 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55fb37440c7948fd90bc20625585998a: Sending DeleteTablet for 3 replicas of tablet 964dc474a5414a49a24d58e639c7ef5f
I20260926 20:21:36.091459 10852 tablet_service.cc:1514] Processing DeleteTablet for tablet 964dc474a5414a49a24d58e639c7ef5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:36 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:58276
I20260926 20:21:36.091920 10986 tablet_service.cc:1514] Processing DeleteTablet for tablet 964dc474a5414a49a24d58e639c7ef5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:36 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:58906
I20260926 20:21:36.091887 11120 tablet_service.cc:1514] Processing DeleteTablet for tablet 964dc474a5414a49a24d58e639c7ef5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:21:36 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:56538
I20260926 20:21:36.092410 11228 tablet_replica.cc:330] T 964dc474a5414a49a24d58e639c7ef5f P 8a17b52b9c4b428ba71cc1ec07657543: stopping tablet replica
I20260926 20:21:36.092696 11228 raft_consensus.cc:2229] T 964dc474a5414a49a24d58e639c7ef5f P 8a17b52b9c4b428ba71cc1ec07657543 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:21:36.093012 11228 raft_consensus.cc:2258] T 964dc474a5414a49a24d58e639c7ef5f P 8a17b52b9c4b428ba71cc1ec07657543 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:21:36.093637 11229 tablet_replica.cc:330] T 964dc474a5414a49a24d58e639c7ef5f P 0cc3a0e3ce974c8e823968b8b7433259: stopping tablet replica
I20260926 20:21:36.093881 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 10789
I20260926 20:21:36.094472 11229 raft_consensus.cc:2229] T 964dc474a5414a49a24d58e639c7ef5f P 0cc3a0e3ce974c8e823968b8b7433259 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:21:36.094792 11228 ts_tablet_manager.cc:1905] T 964dc474a5414a49a24d58e639c7ef5f P 8a17b52b9c4b428ba71cc1ec07657543: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:21:36.095176 11229 raft_consensus.cc:2258] T 964dc474a5414a49a24d58e639c7ef5f P 0cc3a0e3ce974c8e823968b8b7433259 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:21:36.096626 11229 ts_tablet_manager.cc:1905] T 964dc474a5414a49a24d58e639c7ef5f P 0cc3a0e3ce974c8e823968b8b7433259: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:21:36.098378 11228 ts_tablet_manager.cc:1918] T 964dc474a5414a49a24d58e639c7ef5f P 8a17b52b9c4b428ba71cc1ec07657543: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:21:36.098554 11228 log.cc:1196] T 964dc474a5414a49a24d58e639c7ef5f P 8a17b52b9c4b428ba71cc1ec07657543: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/964dc474a5414a49a24d58e639c7ef5f
I20260926 20:21:36.099035 11228 ts_tablet_manager.cc:1939] T 964dc474a5414a49a24d58e639c7ef5f P 8a17b52b9c4b428ba71cc1ec07657543: Deleting consensus metadata
I20260926 20:21:36.100350 11229 ts_tablet_manager.cc:1918] T 964dc474a5414a49a24d58e639c7ef5f P 0cc3a0e3ce974c8e823968b8b7433259: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:21:36.100449 11229 log.cc:1196] T 964dc474a5414a49a24d58e639c7ef5f P 0cc3a0e3ce974c8e823968b8b7433259: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/964dc474a5414a49a24d58e639c7ef5f
I20260926 20:21:36.100623 10711 catalog_manager.cc:4841] TS 8a17b52b9c4b428ba71cc1ec07657543 (127.0.116.194:36645): tablet 964dc474a5414a49a24d58e639c7ef5f (table test-table [id=61a7194b5f194a0f92db1973f8f4432c]) successfully deleted
I20260926 20:21:36.100841 11229 ts_tablet_manager.cc:1939] T 964dc474a5414a49a24d58e639c7ef5f P 0cc3a0e3ce974c8e823968b8b7433259: Deleting consensus metadata
I20260926 20:21:36.102288 10714 catalog_manager.cc:4841] TS 0cc3a0e3ce974c8e823968b8b7433259 (127.0.116.195:40417): tablet 964dc474a5414a49a24d58e639c7ef5f (table test-table [id=61a7194b5f194a0f92db1973f8f4432c]) successfully deleted
W20260926 20:21:36.103709 10713 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.116.193:44131 (error 108)
I20260926 20:21:36.103891 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 10923
I20260926 20:21:36.109025 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:21:36.108669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:44131 (local address 127.0.0.1:58288)
0926 20:21:36.108825 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 20:21:36.108914 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.193:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 20:21:36.109380 10713 catalog_manager.cc:4584] TS a58f038472fe469494e35adb211a8053 (127.0.116.193:44131): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 964dc474a5414a49a24d58e639c7ef5f: Network error: Client connection negotiation failed: client connection to 127.0.116.193:44131: connect: Connection refused (error 111)
I20260926 20:21:36.110378 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 11057
I20260926 20:21:36.116093 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 10695
2026-09-26T20:21:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2354 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:21:36 dist-test-slave-qsm9 krb5kdc[11233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:21:36 dist-test-slave-qsm9 krb5kdc[11233](info): set up 2 sockets
Sep 26 20:21:36 dist-test-slave-qsm9 krb5kdc[11233](info): commencing operation
krb5kdc: starting...
W20260926 20:21:37.153131 467 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:21:37 dist-test-slave-qsm9 krb5kdc[11233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:21:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:21:37Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.116.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.116.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.116.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/rangeradmin_127.0.116.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.116.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/rangeradmin_127.0.116.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.116.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.116.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.116.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/rangerlookup_127.0.116.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.116.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/rangerlookup_127.0.116.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.116.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.116.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.116.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/HTTP_127.0.116.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.116.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/HTTP_127.0.116.212@KRBTEST.COM.keytab.
I20260926 20:21:37.332315 467 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task3m9pMa/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/postgres -l logfile start
2026-09-26 20:21:40.015 UTC [11263] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 20:21:40.015 UTC [11263] LOG: listening on IPv4 address "127.0.116.212", port 41727
2026-09-26 20:21:40.019 UTC [11263] LOG: listening on Unix socket "/tmp/.s.PGSQL.41727"
2026-09-26 20:21:40.050 UTC [11266] LOG: database system was shut down at 2026-09-26 20:21:38 UTC
2026-09-26 20:21:40.054 UTC [11263] LOG: database system is ready to accept connections
I20260926 20:21:41.020757 467 mini_postgres.cc:96] Postgres bound to 41727
2026-09-26 20:21:41.027 UTC [11274] FATAL: database "slave" does not exist
127.0.116.212:41727 - accepting connections
I20260926 20:21:41.028026 467 mini_ranger.cc:161] Starting Ranger...
I20260926 20:21:41.043614 467 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 20:21:41.601716 467 mini_ranger.cc:87] Created ranger Postgres database
I20260926 20:21:41.601840 467 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ranger-admin
2026-09-26 20:21:41,910 [I] DB FLAVOR :POSTGRES
2026-09-26 20:21:41,911 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:21:41,911 [I] Checking connection..
2026-09-26 20:21:41,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 20:21:42,181 [I] Checking connection passed.
2026-09-26 20:21:42,182 [I] --------- Verifying version history table ---------
2026-09-26 20:21:42,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:21:42,456 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:21:42,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:21:42,709 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:21:42,710 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 20:21:42,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 20:21:42.973 UTC [11338] WARNING: there is no transaction in progress
2026-09-26 20:21:42,980 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 20:21:42,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:21:43,233 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 20:21:43,233 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:21:43,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 20:21:43,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 20:21:43,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qsm9.c.gcp-upstream.internal','N') ;"
2026-09-26 20:21:43,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:21:44,200 [I] Table x_portal_user does not exist in database ranger
2026-09-26 20:21:44,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:21:44,457 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 20:21:44,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:21:44,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:21:44,973 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 20:21:44,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 20:21:45.981 UTC [11446] WARNING: there is no transaction in progress
2026-09-26 20:21:45.999 UTC [11446] WARNING: there is no transaction in progress
2026-09-26 20:21:46.016 UTC [11446] WARNING: there is no transaction in progress
2026-09-26 20:21:46.033 UTC [11446] WARNING: there is no transaction in progress
2026-09-26 20:21:46.050 UTC [11446] WARNING: there is no transaction in progress
2026-09-26 20:21:46.068 UTC [11446] WARNING: there is no transaction in progress
2026-09-26 20:21:46.128 UTC [11446] WARNING: there is no transaction in progress
2026-09-26 20:21:46.136 UTC [11446] WARNING: there is no transaction in progress
2026-09-26 20:21:46.155 UTC [11446] WARNING: there is no transaction in progress
2026-09-26 20:21:46.168 UTC [11446] WARNING: there is no transaction in progress
2026-09-26 20:21:46.180 UTC [11446] WARNING: there is no transaction in progress
2026-09-26 20:21:46.196 UTC [11446] WARNING: there is no transaction in progress
2026-09-26 20:21:46.206 UTC [11446] WARNING: there is no transaction in progress
2026-09-26 20:21:46.212 UTC [11446] WARNING: there is no transaction in progress
2026-09-26 20:21:46.219 UTC [11446] WARNING: there is no transaction in progress
2026-09-26 20:21:46.227 UTC [11446] WARNING: there is no transaction in progress
2026-09-26 20:21:46,903 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 20:21:46,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 20:21:47,134 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 20:21:47,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:21:47,411 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 20:21:47,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:21:47,661 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 20:21:47,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:21:47,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:21:48,160 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qsm9.c.gcp-upstream.internal';"
2026-09-26 20:21:48,398 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 20:21:48,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 20:21:48,646 [I] DB_PATCHES have already been applied
I20260926 20:21:48.651681 467 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ranger-admin:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task3m9pMa/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 20:21:48.651748 467 mini_ranger.cc:193] Using host: 127.0.116.212
I20260926 20:21:48.655238 467 mini_ranger.cc:227] Ranger admin URL: http://127.0.116.212:52077
Sep 26, 2026 8:21:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:21:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:21:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 8:21:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 8:21:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 8:21:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 8:21:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 20:21:49.771733 467 mini_ranger.cc:160] Time spent starting Ranger: real 8.744s user 0.000s sys 0.011s
Sep 26, 2026 8:22:09 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 8:22:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 8:22:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 8:22:09 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 8:22:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 8:22:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 8:22:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 8:22:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 8:22:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 8:22:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 8:22:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 8:22:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 8:22:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 8:22:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 8:22:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 8:22:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 8:22:09 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 8:22:10 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 20:22:10.785624 467 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.116.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.116.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.116.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/rangerkms_127.0.116.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.116.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/rangerkms_127.0.116.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.116.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.116.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.116.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/HTTP_127.0.116.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.116.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/HTTP_127.0.116.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 20:22:10 dist-test-slave-qsm9 krb5kdc[11233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454130, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 20:22:10.891474 467 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 20:22:10.902151 467 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 20:22:11.937723 467 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 20:22:11.937842 467 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ranger-kms
I20260926 20:22:11.937961 467 mini_ranger_kms.cc:227] Using postgres at 127.0.116.212:41727
2026-09-26 20:22:12,126 [I] DB FLAVOR :POSTGRES
2026-09-26 20:22:12,127 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:22:12,127 [I] Checking connection
2026-09-26 20:22:12,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 20:22:12,382 [I] connection success
2026-09-26 20:22:12,382 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 20:22:12,382 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 20:22:12,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 20:22:12,651 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 20:22:12,651 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:22:12,652 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 20:22:12,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:41727/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 20:22:12,908 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 20:22:13.376963 467 mini_ranger_kms.cc:310] Created kms service
I20260926 20:22:13.570914 467 mini_ranger_kms.cc:326] Created kudu user
I20260926 20:22:13.612689 467 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 20:22:13 dist-test-slave-qsm9 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454130, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
I20260926 20:22:14.034816 467 mini_ranger_kms.cc:384] Added ranger policy
I20260926 20:22:14.034974 467 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ranger-kms:/tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 20:22:14.035000 467 mini_ranger_kms.cc:240] Using host: 127.0.116.212
I20260926 20:22:14.038656 467 mini_ranger_kms.cc:276] Ranger KMS PID: 11626
I20260926 20:22:14.038794 467 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.116.212:48145
Sep 26, 2026 8:22:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:22:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:22:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 8:22:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 8:22:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 8:22:15 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 20:22:16.360163 467 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.469s user 0.029s sys 0.143s
I20260926 20:22:16.360291 467 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 20:22:16.360391 467 mini_ranger_kms.cc:401] 127.0.116.212:48145/kms/v1/keys
Sep 26 20:22:16 dist-test-slave-qsm9 krb5kdc[11233](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790454136, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.116.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 20:22:17 dist-test-slave-qsm9 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454136, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.116.212@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
Sep 26 20:22:17 dist-test-slave-qsm9 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454136, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.116.212@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
Sep 26 20:22:19 dist-test-slave-qsm9 krb5kdc[11233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:22:19.961335 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:35699
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:35699
--ranger_config_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.116.212:48145/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:22:20.077667 11677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:22:20.077994 11677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:22:20.078037 11677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:22:20.082242 11677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:22:20.082311 11677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:22:20.082331 11677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:22:20.082348 11677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:22:20.082363 11677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:22:20.088351 11677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.116.212:48145/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:35699
--ranger_config_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:35699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:22:20.089543 11677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:22:20.090641 11677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:22:20.099614 11683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:22:20.099614 11685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:22:20.099625 11682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:22:20.099777 11677 server_base.cc:1034] running on GCE node
I20260926 20:22:20.100888 11677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:22:20.102119 11677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:22:20.103303 11677 hybrid_clock.cc:648] HybridClock initialized: now 1790454140103282 us; error 43 us; skew 500 ppm
Sep 26 20:22:20 dist-test-slave-qsm9 krb5kdc[11233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:22:20.106185 11677 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:22:20.107211 11677 webserver.cc:460] Webserver started at http://127.0.116.254:38253/ using document root <none> and password file <none>
I20260926 20:22:20.107829 11677 fs_manager.cc:359] Metadata directory not provided
I20260926 20:22:20.107893 11677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:22:20.108166 11677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:22:20 dist-test-slave-qsm9 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
I20260926 20:22:20.156446 11677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "41b9ca6578f94b42a2fbd3ae01a6516c"
format_stamp: "Formatted at 2026-09-26 20:22:20 on dist-test-slave-qsm9"
server_key: "4717646455dcd382e87c126b3e7ea644"
server_key_iv: "1fe0f7ff09d30cc3d8263cc761a477df"
server_key_version: "kuduclusterkey@0"
I20260926 20:22:20.157120 11677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "41b9ca6578f94b42a2fbd3ae01a6516c"
format_stamp: "Formatted at 2026-09-26 20:22:20 on dist-test-slave-qsm9"
server_key: "4717646455dcd382e87c126b3e7ea644"
server_key_iv: "1fe0f7ff09d30cc3d8263cc761a477df"
server_key_version: "kuduclusterkey@0"
I20260926 20:22:20.161435 11677 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 20:22:20.176011 11692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:22:20.177495 11677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:22:20.177613 11677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "41b9ca6578f94b42a2fbd3ae01a6516c"
format_stamp: "Formatted at 2026-09-26 20:22:20 on dist-test-slave-qsm9"
server_key: "4717646455dcd382e87c126b3e7ea644"
server_key_iv: "1fe0f7ff09d30cc3d8263cc761a477df"
server_key_version: "kuduclusterkey@0"
I20260926 20:22:20.177778 11677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:22:20.192612 11677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:22:20.195937 11677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:22:20.196149 11677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:22:20.205309 11677 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:35699
I20260926 20:22:20.205335 11744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:35699 every 8 connection(s)
I20260926 20:22:20.206444 11677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:22:20.209473 11745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:22:20.210068 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 11677
I20260926 20:22:20.210207 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:22:20.210541 467 external_mini_cluster.cc:1428] Setting key 6d3d4e4e7ff6f9a8c256384114548c6e
I20260926 20:22:20.216792 11745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c: Bootstrap starting.
I20260926 20:22:20.219347 11745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c: Neither blocks nor log segments found. Creating new log.
I20260926 20:22:20.220220 11745 log.cc:824] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:22:20 dist-test-slave-qsm9 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:22:20.223349 11745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c: No bootstrap required, opened a new log
I20260926 20:22:20.227653 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:22:20.212177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34488 (local address 127.0.116.254:35699)
0926 20:22:20.212671 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0926 20:22:20.212690 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:22:20.212737 (+ 47us) server_negotiation.cc:407] Connection header received
0926 20:22:20.213596 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:22:20.213633 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:22:20.214095 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:22:20.214473 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:22:20.215433 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:22:20.216913 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:22:20.218183 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:22:20.218840 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:22:20.219005 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:22:20.222142 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:22:20.222148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:22:20.222165 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:22:20.222193 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:22:20.224542 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:22:20.225033 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:22:20.225040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:22:20.225046 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:22:20.225159 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:22:20.225456 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:22:20.225459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:22:20.225460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:22:20.225866 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:22:20.226074 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 20:22:20.226377 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 20:22:20.226790 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":140,"threads_started":1}
I20260926 20:22:20.227658 11745 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b9ca6578f94b42a2fbd3ae01a6516c" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 35699 } }
I20260926 20:22:20.228005 11745 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:22:20.228075 11745 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41b9ca6578f94b42a2fbd3ae01a6516c, State: Initialized, Role: FOLLOWER
I20260926 20:22:20.228593 11745 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b9ca6578f94b42a2fbd3ae01a6516c" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 35699 } }
I20260926 20:22:20.228752 11745 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:22:20.228839 11745 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:22:20.228945 11745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:22:20.230151 11745 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b9ca6578f94b42a2fbd3ae01a6516c" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 35699 } }
I20260926 20:22:20.230535 11745 leader_election.cc:302] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41b9ca6578f94b42a2fbd3ae01a6516c; no voters:
I20260926 20:22:20.230885 11745 leader_election.cc:288] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:22:20.230989 11750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:22:20.231266 11750 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c [term 1 LEADER]: Becoming Leader. State: Replica: 41b9ca6578f94b42a2fbd3ae01a6516c, State: Running, Role: LEADER
I20260926 20:22:20.231650 11750 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b9ca6578f94b42a2fbd3ae01a6516c" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 35699 } }
I20260926 20:22:20.232209 11745 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:22:20.233762 11752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41b9ca6578f94b42a2fbd3ae01a6516c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b9ca6578f94b42a2fbd3ae01a6516c" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 35699 } } }
I20260926 20:22:20.233888 11752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c [sys.catalog]: This master's current role is: LEADER
I20260926 20:22:20.234129 11752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41b9ca6578f94b42a2fbd3ae01a6516c. Latest consensus state: current_term: 1 leader_uuid: "41b9ca6578f94b42a2fbd3ae01a6516c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b9ca6578f94b42a2fbd3ae01a6516c" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 35699 } } }
I20260926 20:22:20.234198 11752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c [sys.catalog]: This master's current role is: LEADER
I20260926 20:22:20.234823 11745 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 20:22:20.235769 11759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:22:20.238354 11759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:22:20.248020 11759 catalog_manager.cc:1347] Generated new cluster ID: 8dc44690a1ad436e924fb7d652ea461a
I20260926 20:22:20.248170 11759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:22:20.291676 11759 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:22:20.293131 11759 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:22:20.298664 11759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c: Generated new TSK 0
I20260926 20:22:20.299687 11759 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
I20260926 20:22:21.071933 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.116.212:48145/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:35699
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:22:21.197882 11790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:22:21.198239 11790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:22:21.198336 11790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:22:21.202692 11790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:22:21.202790 11790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:22:21.202889 11790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:22:21.210204 11790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.116.212:48145/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:35699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:22:21.211624 11790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:22:21.212950 11790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:22:21.224854 11796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:22:21.225322 11798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:22:21.225389 11795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:22:21.227208 11790 server_base.cc:1034] running on GCE node
I20260926 20:22:21.227792 11790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:22:21.228674 11790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:22:21.231976 11790 hybrid_clock.cc:648] HybridClock initialized: now 1790454141231934 us; error 54 us; skew 500 ppm
Sep 26 20:22:21 dist-test-slave-qsm9 krb5kdc[11233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:22:21.235140 11790 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:22:21.236253 11790 webserver.cc:460] Webserver started at http://127.0.116.193:35999/ using document root <none> and password file <none>
I20260926 20:22:21.236868 11790 fs_manager.cc:359] Metadata directory not provided
I20260926 20:22:21.236969 11790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:22:21.237232 11790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:22:21 dist-test-slave-qsm9 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
I20260926 20:22:21.253517 11790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "5dd53dc83fe74e8fbfe34f52687c775d"
format_stamp: "Formatted at 2026-09-26 20:22:21 on dist-test-slave-qsm9"
server_key: "af901aec3c39c6ead01ecc636bccd57c"
server_key_iv: "6c890de1f352078381ac11d794c520f6"
server_key_version: "kuduclusterkey@0"
I20260926 20:22:21.254390 11790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "5dd53dc83fe74e8fbfe34f52687c775d"
format_stamp: "Formatted at 2026-09-26 20:22:21 on dist-test-slave-qsm9"
server_key: "af901aec3c39c6ead01ecc636bccd57c"
server_key_iv: "6c890de1f352078381ac11d794c520f6"
server_key_version: "kuduclusterkey@0"
I20260926 20:22:21.259708 11790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 20:22:21.283098 11807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:22:21.285035 11790 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:22:21.285193 11790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "5dd53dc83fe74e8fbfe34f52687c775d"
format_stamp: "Formatted at 2026-09-26 20:22:21 on dist-test-slave-qsm9"
server_key: "af901aec3c39c6ead01ecc636bccd57c"
server_key_iv: "6c890de1f352078381ac11d794c520f6"
server_key_version: "kuduclusterkey@0"
I20260926 20:22:21.285360 11790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:22:21.312191 11790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:22:21.316243 11790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:22:21.316514 11790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:22:21.317341 11790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:22:21.318472 11790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:22:21.318589 11790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:22:21.318717 11790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:22:21.318797 11790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:22:21.332825 11790 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:40331
I20260926 20:22:21.334220 11790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:22:21.334996 11921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:40331 every 8 connection(s)
I20260926 20:22:21.342542 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 11790
I20260926 20:22:21.342669 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:22:21.343034 467 external_mini_cluster.cc:1428] Setting key 85ba30c61613ecc0fa34e64941e6ff56
WARNING: no policy specified for kudu/127.0.116.194@KRBTEST.COM; defaulting to no policy
Sep 26 20:22:21 dist-test-slave-qsm9 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.194@KRBTEST.COM" created.
I20260926 20:22:21.367568 11925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:22:21.337950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:57935 (local address 127.0.116.254:35699)
0926 20:22:21.338213 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 20:22:21.338218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:22:21.339614 (+ 1396us) server_negotiation.cc:407] Connection header received
0926 20:22:21.340543 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:22:21.340547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:22:21.340610 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:22:21.340698 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:22:21.341949 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:22:21.342744 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:22:21.357361 (+ 14617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:22:21.357758 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:22:21.357901 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:22:21.362073 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:22:21.362078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:22:21.362081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:22:21.362121 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:22:21.364223 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:22:21.365014 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:22:21.365018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:22:21.365020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:22:21.365119 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:22:21.366028 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:22:21.366032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:22:21.366033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:22:21.366302 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:22:21.366398 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:22:21.367193 (+ 795us) server_negotiation.cc:299] Negotiation successful
0926 20:22:21.367343 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":100,"threads_started":1}
I20260926 20:22:21.369362 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:22:21.337862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:35699 (local address 127.0.116.193:57935)
0926 20:22:21.339016 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0926 20:22:21.339060 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:22:21.340321 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:22:21.340885 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:22:21.340904 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:22:21.341218 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:22:21.341789 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:22:21.341814 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:22:21.342921 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:22:21.342925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:22:21.357114 (+ 14189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:22:21.357141 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:22:21.358140 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:22:21.358147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:22:21.358452 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:22:21.359281 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:22:21.359304 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:22:21.361788 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:22:21.364418 (+ 2630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:22:21.364425 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:22:21.364447 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:22:21.364877 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:22:21.365671 (+ 794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:22:21.365674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:22:21.365679 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:22:21.365906 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:22:21.366484 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:22:21.366491 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:22:21.366857 (+ 366us) client_negotiation.cc:769] Sending connection context
0926 20:22:21.367859 (+ 1002us) client_negotiation.cc:240] Negotiation successful
0926 20:22:21.368250 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"thread_start_us":139,"threads_started":1}
I20260926 20:22:21.371248 11922 heartbeater.cc:347] Connected to a master server at 127.0.116.254:35699
I20260926 20:22:21.371554 11922 heartbeater.cc:464] Registering TS with master...
I20260926 20:22:21.372219 11922 heartbeater.cc:511] Master 127.0.116.254:35699 requested a full tablet report, sending...
I20260926 20:22:21.374254 11709 ts_manager.cc:194] Registered new tserver with Master: 5dd53dc83fe74e8fbfe34f52687c775d (127.0.116.193:40331)
I20260926 20:22:21.375893 11709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:57935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
I20260926 20:22:21.384999 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.116.212:48145/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:35699
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:22:21 dist-test-slave-qsm9 krb5kdc[11233](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790454141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 20:22:21.550873 11929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:22:21.551275 11929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:22:21.551350 11929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:22:21.557533 11929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:22:21.557629 11929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:22:21.557718 11929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:22:21.565943 11929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.116.212:48145/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:35699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:22:21.567638 11929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:22:21.569059 11929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:22:21.581974 11935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:22:21.582168 11936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:22:21.582051 11938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:22:21.582515 11929 server_base.cc:1034] running on GCE node
I20260926 20:22:21.583141 11929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:22:21.584007 11929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:22:21.585196 11929 hybrid_clock.cc:648] HybridClock initialized: now 1790454141585172 us; error 41 us; skew 500 ppm
Sep 26 20:22:21 dist-test-slave-qsm9 krb5kdc[11233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:22:21.588850 11929 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:22:21.589967 11929 webserver.cc:460] Webserver started at http://127.0.116.194:34371/ using document root <none> and password file <none>
I20260926 20:22:21.590719 11929 fs_manager.cc:359] Metadata directory not provided
I20260926 20:22:21.590808 11929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:22:21.591049 11929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:22:21 dist-test-slave-qsm9 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
I20260926 20:22:21.610087 11929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "4d32e51bc30d4fbc8b931d8b792190e9"
format_stamp: "Formatted at 2026-09-26 20:22:21 on dist-test-slave-qsm9"
server_key: "106267eee7a96e653cd96b1d9f91f076"
server_key_iv: "628fb279d33d60777c0d79724c7b916c"
server_key_version: "kuduclusterkey@0"
I20260926 20:22:21.610968 11929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "4d32e51bc30d4fbc8b931d8b792190e9"
format_stamp: "Formatted at 2026-09-26 20:22:21 on dist-test-slave-qsm9"
server_key: "106267eee7a96e653cd96b1d9f91f076"
server_key_iv: "628fb279d33d60777c0d79724c7b916c"
server_key_version: "kuduclusterkey@0"
I20260926 20:22:21.617079 11929 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260926 20:22:21.643926 11945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:22:21.645821 11929 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.001s
I20260926 20:22:21.646044 11929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "4d32e51bc30d4fbc8b931d8b792190e9"
format_stamp: "Formatted at 2026-09-26 20:22:21 on dist-test-slave-qsm9"
server_key: "106267eee7a96e653cd96b1d9f91f076"
server_key_iv: "628fb279d33d60777c0d79724c7b916c"
server_key_version: "kuduclusterkey@0"
I20260926 20:22:21.646283 11929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:22:21.666597 11929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:22:21.670848 11929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:22:21.671070 11929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:22:21.671931 11929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:22:21.673105 11929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:22:21.673219 11929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:22:21.673317 11929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:22:21.673383 11929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:22:21.685649 11929 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:38797
I20260926 20:22:21.685724 12058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:38797 every 8 connection(s)
I20260926 20:22:21.686666 11929 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:22:21.690225 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 11929
I20260926 20:22:21.690376 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:22:21.690716 467 external_mini_cluster.cc:1428] Setting key 3a484dc4cd83444f16f34137b5bbda5c
WARNING: no policy specified for kudu/127.0.116.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.195@KRBTEST.COM" created.
Sep 26 20:22:21 dist-test-slave-qsm9 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:22:21.717864 11925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:22:21.688742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:50927 (local address 127.0.116.254:35699)
0926 20:22:21.688895 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:22:21.688901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:22:21.689882 (+ 981us) server_negotiation.cc:407] Connection header received
0926 20:22:21.694502 (+ 4620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:22:21.694508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:22:21.694575 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:22:21.694702 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:22:21.696751 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:22:21.697662 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:22:21.699362 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:22:21.699792 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:22:21.699910 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:22:21.704920 (+ 5010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:22:21.704923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:22:21.704925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:22:21.704954 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:22:21.706705 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:22:21.707609 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:22:21.707612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:22:21.707613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:22:21.707691 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:22:21.708574 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:22:21.708587 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:22:21.708592 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:22:21.708875 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:22:21.708987 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:22:21.717479 (+ 8492us) server_negotiation.cc:299] Negotiation successful
0926 20:22:21.717677 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:22:21.719825 12061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:22:21.689156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:35699 (local address 127.0.116.194:50927)
0926 20:22:21.689686 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 20:22:21.689749 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 20:22:21.692180 (+ 2431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:22:21.695324 (+ 3144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:22:21.695335 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:22:21.695751 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:22:21.696569 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:22:21.696592 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:22:21.697821 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:22:21.697824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:22:21.699208 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:22:21.699221 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:22:21.700017 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:22:21.700025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:22:21.700286 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:22:21.701503 (+ 1217us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:22:21.701530 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:22:21.704695 (+ 3165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:22:21.706862 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:22:21.706872 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:22:21.706894 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:22:21.707445 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:22:21.707787 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:22:21.707791 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:22:21.707795 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:22:21.708434 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:22:21.709013 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:22:21.709020 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:22:21.717048 (+ 8028us) client_negotiation.cc:769] Sending connection context
0926 20:22:21.718087 (+ 1039us) client_negotiation.cc:240] Negotiation successful
0926 20:22:21.718545 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":115,"threads_started":1}
I20260926 20:22:21.722702 12059 heartbeater.cc:347] Connected to a master server at 127.0.116.254:35699
I20260926 20:22:21.723011 12059 heartbeater.cc:464] Registering TS with master...
I20260926 20:22:21.723703 12059 heartbeater.cc:511] Master 127.0.116.254:35699 requested a full tablet report, sending...
I20260926 20:22:21.724932 11709 ts_manager.cc:194] Registered new tserver with Master: 4d32e51bc30d4fbc8b931d8b792190e9 (127.0.116.194:38797)
I20260926 20:22:21.725545 11709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:50927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
I20260926 20:22:21.739475 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.116.212:48145/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:35699
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:22:21.879570 12064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:22:21.880081 12064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:22:21.880225 12064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Sep 26 20:22:21 dist-test-slave-qsm9 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
W20260926 20:22:21.886793 12064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:22:21.886922 12064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:22:21.887046 12064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:22:21.895320 12064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.116.212:48145/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:35699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:22:21.897063 12064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:22:21.898494 12064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:22:21.912645 12072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:22:21.913116 12070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:22:21.913717 12064 server_base.cc:1034] running on GCE node
W20260926 20:22:21.913836 12069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:22:21.914456 12064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:22:21.915354 12064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:22:21.930889 12064 hybrid_clock.cc:648] HybridClock initialized: now 1790454141930762 us; error 142 us; skew 500 ppm
Sep 26 20:22:21 dist-test-slave-qsm9 krb5kdc[11233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:22:21.935346 12064 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:22:21.936760 12064 webserver.cc:460] Webserver started at http://127.0.116.195:45003/ using document root <none> and password file <none>
I20260926 20:22:21.937393 12064 fs_manager.cc:359] Metadata directory not provided
I20260926 20:22:21.937497 12064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:22:21.937860 12064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:22:21 dist-test-slave-qsm9 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
I20260926 20:22:21.958173 12064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "cae22c0059ad4d0d8dd22e1cc9b23f49"
format_stamp: "Formatted at 2026-09-26 20:22:21 on dist-test-slave-qsm9"
server_key: "b2cca29dc5ac128bc3e72bc22723b5b8"
server_key_iv: "fb7b0b53a555f985f58058f56dfa9dec"
server_key_version: "kuduclusterkey@0"
I20260926 20:22:21.960711 12064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "cae22c0059ad4d0d8dd22e1cc9b23f49"
format_stamp: "Formatted at 2026-09-26 20:22:21 on dist-test-slave-qsm9"
server_key: "b2cca29dc5ac128bc3e72bc22723b5b8"
server_key_iv: "fb7b0b53a555f985f58058f56dfa9dec"
server_key_version: "kuduclusterkey@0"
I20260926 20:22:21.968886 12064 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260926 20:22:21.988299 12079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:22:21.989992 12064 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260926 20:22:21.990159 12064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "cae22c0059ad4d0d8dd22e1cc9b23f49"
format_stamp: "Formatted at 2026-09-26 20:22:21 on dist-test-slave-qsm9"
server_key: "b2cca29dc5ac128bc3e72bc22723b5b8"
server_key_iv: "fb7b0b53a555f985f58058f56dfa9dec"
server_key_version: "kuduclusterkey@0"
I20260926 20:22:21.990319 12064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:22:22.016392 12064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:22:22.020282 12064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:22:22.020521 12064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:22:22.021224 12064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:22:22.022244 12064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:22:22.022312 12064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:22:22.022367 12064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:22:22.022521 12064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 20:22:22.034699 12064 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:35383
I20260926 20:22:22.035089 12192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:35383 every 8 connection(s)
I20260926 20:22:22.036135 12064 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:22:22.044912 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 12064
I20260926 20:22:22.045046 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:22:22.045368 467 external_mini_cluster.cc:1428] Setting key 98e688b7ef8638a1e9cd01e80d099f92
Sep 26 20:22:22 dist-test-slave-qsm9 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:22:22.052374 11925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:22:22.040169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:34593 (local address 127.0.116.254:35699)
0926 20:22:22.040315 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 20:22:22.040320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:22:22.040784 (+ 464us) server_negotiation.cc:407] Connection header received
0926 20:22:22.041740 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:22:22.041744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:22:22.041802 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:22:22.041908 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:22:22.043005 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.043596 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.044804 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.045164 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.045251 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:22:22.048487 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:22:22.048493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:22:22.048499 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:22:22.048526 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:22:22.050144 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:22:22.050674 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:22:22.050679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:22:22.050681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:22:22.050761 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:22:22.051105 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:22:22.051109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:22:22.051111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:22:22.051351 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:22:22.051458 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 20:22:22.052050 (+ 592us) server_negotiation.cc:299] Negotiation successful
0926 20:22:22.052204 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:22:22.053408 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:22:22.038872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:35699 (local address 127.0.116.195:34593)
0926 20:22:22.040604 (+ 1732us) negotiation.cc:107] Waiting for socket to connect
0926 20:22:22.040665 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 20:22:22.041524 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:22:22.042091 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:22:22.042098 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:22:22.042367 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:22:22.042848 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:22:22.042861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.043719 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.043722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:22:22.044654 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:22:22.044666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.045467 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.045472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:22:22.045670 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:22:22.046618 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:22:22.046642 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:22:22.048319 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:22:22.050278 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:22:22.050284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:22:22.050309 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:22:22.050571 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:22:22.050858 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:22:22.050861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:22:22.050862 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:22:22.051009 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:22:22.051459 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:22:22.051467 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:22:22.051784 (+ 317us) client_negotiation.cc:769] Sending connection context
0926 20:22:22.052011 (+ 227us) client_negotiation.cc:240] Negotiation successful
0926 20:22:22.052399 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1471,"thread_start_us":167,"threads_started":1}
I20260926 20:22:22.054591 12193 heartbeater.cc:347] Connected to a master server at 127.0.116.254:35699
I20260926 20:22:22.054847 12193 heartbeater.cc:464] Registering TS with master...
I20260926 20:22:22.055369 12193 heartbeater.cc:511] Master 127.0.116.254:35699 requested a full tablet report, sending...
I20260926 20:22:22.056450 11709 ts_manager.cc:194] Registered new tserver with Master: cae22c0059ad4d0d8dd22e1cc9b23f49 (127.0.116.195:35383)
I20260926 20:22:22.057046 11709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:34593
I20260926 20:22:22.060431 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:22:22.070375 11925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:22:22.062480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41316 (local address 127.0.116.254:35699)
0926 20:22:22.062608 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 20:22:22.062613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:22:22.062864 (+ 251us) server_negotiation.cc:407] Connection header received
0926 20:22:22.063023 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:22:22.063027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:22:22.063079 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:22:22.063161 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:22:22.063966 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.064488 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.065404 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.065677 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.065790 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:22:22.067192 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:22:22.067196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:22:22.067199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:22:22.067224 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:22:22.068736 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:22:22.069157 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:22:22.069163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:22:22.069165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:22:22.069248 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:22:22.069521 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:22:22.069526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:22:22.069528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:22:22.069708 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:22:22.069844 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 20:22:22.070079 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 20:22:22.070193 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:22:22.073673 11709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:22:22.076957 11709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:22:22.094538 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:22:22.087513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:40331 (local address 127.0.0.1:55444)
0926 20:22:22.088980 (+ 1467us) negotiation.cc:107] Waiting for socket to connect
0926 20:22:22.088994 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:22:22.089068 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:22:22.089757 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:22:22.089760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:22:22.089774 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:22:22.089890 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:22:22.089897 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.091329 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.091332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:22:22.092695 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:22:22.092705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.093960 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.093964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:22:22.094091 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:22:22.094137 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 20:22:22.094226 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 20:22:22.094356 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1388,"mutex_wait_us":13,"thread_start_us":86,"threads_started":1}
I20260926 20:22:22.095362 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:22:22.085859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:38797 (local address 127.0.0.1:44898)
0926 20:22:22.087859 (+ 2000us) negotiation.cc:107] Waiting for socket to connect
0926 20:22:22.087903 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:22:22.088036 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:22:22.088731 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:22:22.088737 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:22:22.088759 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:22:22.088894 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:22:22.088901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.091149 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.091152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:22:22.093505 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:22:22.093516 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.094908 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.094911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:22:22.095025 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:22:22.095056 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 20:22:22.095128 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 20:22:22.095209 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1960,"mutex_wait_us":63,"spinlock_wait_cycles":3456,"thread_start_us":68,"threads_started":1}
I20260926 20:22:22.095837 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:22:22.086625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44898 (local address 127.0.116.194:38797)
0926 20:22:22.088311 (+ 1686us) server_negotiation.cc:206] Beginning negotiation
0926 20:22:22.088320 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:22:22.088341 (+ 21us) server_negotiation.cc:407] Connection header received
0926 20:22:22.088401 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:22:22.088405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:22:22.088579 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:22:22.088732 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:22:22.089041 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.091023 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.093705 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.094509 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.094617 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:22:22.094851 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 20:22:22.095582 (+ 731us) server_negotiation.cc:299] Negotiation successful
0926 20:22:22.095693 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1599,"thread_start_us":107,"threads_started":1}
I20260926 20:22:22.098738 11985 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea953f468bb4a1a964dbcb83996c0d0 (DEFAULT_TABLE table=test-table [id=b28841077d7f410cbf50b1894957d27b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:22:22.099781 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:22:22.087661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55444 (local address 127.0.116.193:40331)
0926 20:22:22.089298 (+ 1637us) server_negotiation.cc:206] Beginning negotiation
0926 20:22:22.089305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:22:22.089327 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:22:22.089398 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:22:22.089403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:22:22.089615 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:22:22.090043 (+ 428us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:22:22.090195 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.091220 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.092850 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.093792 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.095323 (+ 1531us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:22:22.095819 (+ 496us) server_negotiation.cc:1035] Waiting for connection context
0926 20:22:22.095877 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 20:22:22.099640 (+ 3763us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1538,"thread_start_us":111,"threads_started":1}
I20260926 20:22:22.099911 11985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea953f468bb4a1a964dbcb83996c0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:22:22.103045 11853 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea953f468bb4a1a964dbcb83996c0d0 (DEFAULT_TABLE table=test-table [id=b28841077d7f410cbf50b1894957d27b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:22:22.104203 11853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea953f468bb4a1a964dbcb83996c0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:22:22.105145 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:22:22.085742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:35383 (local address 127.0.0.1:58780)
0926 20:22:22.087849 (+ 2107us) negotiation.cc:107] Waiting for socket to connect
0926 20:22:22.087903 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 20:22:22.088036 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:22:22.095102 (+ 7066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:22:22.095105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:22:22.095125 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:22:22.095305 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:22:22.095312 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.102286 (+ 6974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.102290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:22:22.103455 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:22:22.103472 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.104700 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.104706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:22:22.104810 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:22:22.104845 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 20:22:22.104906 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 20:22:22.104991 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1926,"spinlock_wait_cycles":768,"thread_start_us":137,"threads_started":1}
I20260926 20:22:22.106112 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:22:22.086646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58780 (local address 127.0.116.195:35383)
0926 20:22:22.094313 (+ 7667us) server_negotiation.cc:206] Beginning negotiation
0926 20:22:22.094324 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:22:22.094353 (+ 29us) server_negotiation.cc:407] Connection header received
0926 20:22:22.094520 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:22:22.094526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:22:22.094751 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:22:22.094930 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:22:22.100907 (+ 5977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.102109 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.103701 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.104530 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.105669 (+ 1139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:22:22.105823 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 20:22:22.105876 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 20:22:22.105971 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7544,"thread_start_us":156,"threads_started":1}
I20260926 20:22:22.109339 12117 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea953f468bb4a1a964dbcb83996c0d0 (DEFAULT_TABLE table=test-table [id=b28841077d7f410cbf50b1894957d27b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:22:22.110417 12117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea953f468bb4a1a964dbcb83996c0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:22:22.119221 12213 tablet_bootstrap.cc:492] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9: Bootstrap starting.
I20260926 20:22:22.120146 12212 tablet_bootstrap.cc:492] T 1ea953f468bb4a1a964dbcb83996c0d0 P cae22c0059ad4d0d8dd22e1cc9b23f49: Bootstrap starting.
I20260926 20:22:22.122255 12213 tablet_bootstrap.cc:654] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9: Neither blocks nor log segments found. Creating new log.
I20260926 20:22:22.123159 12211 tablet_bootstrap.cc:492] T 1ea953f468bb4a1a964dbcb83996c0d0 P 5dd53dc83fe74e8fbfe34f52687c775d: Bootstrap starting.
I20260926 20:22:22.123656 12213 log.cc:824] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9: Log is configured to *not* fsync() on all Append() calls
I20260926 20:22:22.124301 12212 tablet_bootstrap.cc:654] T 1ea953f468bb4a1a964dbcb83996c0d0 P cae22c0059ad4d0d8dd22e1cc9b23f49: Neither blocks nor log segments found. Creating new log.
I20260926 20:22:22.126001 12211 tablet_bootstrap.cc:654] T 1ea953f468bb4a1a964dbcb83996c0d0 P 5dd53dc83fe74e8fbfe34f52687c775d: Neither blocks nor log segments found. Creating new log.
I20260926 20:22:22.126721 12211 log.cc:824] T 1ea953f468bb4a1a964dbcb83996c0d0 P 5dd53dc83fe74e8fbfe34f52687c775d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:22:22.127952 12212 log.cc:824] T 1ea953f468bb4a1a964dbcb83996c0d0 P cae22c0059ad4d0d8dd22e1cc9b23f49: Log is configured to *not* fsync() on all Append() calls
I20260926 20:22:22.129580 12211 tablet_bootstrap.cc:492] T 1ea953f468bb4a1a964dbcb83996c0d0 P 5dd53dc83fe74e8fbfe34f52687c775d: No bootstrap required, opened a new log
I20260926 20:22:22.129813 12211 ts_tablet_manager.cc:1397] T 1ea953f468bb4a1a964dbcb83996c0d0 P 5dd53dc83fe74e8fbfe34f52687c775d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 20:22:22.131119 12212 tablet_bootstrap.cc:492] T 1ea953f468bb4a1a964dbcb83996c0d0 P cae22c0059ad4d0d8dd22e1cc9b23f49: No bootstrap required, opened a new log
I20260926 20:22:22.131402 12212 ts_tablet_manager.cc:1397] T 1ea953f468bb4a1a964dbcb83996c0d0 P cae22c0059ad4d0d8dd22e1cc9b23f49: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260926 20:22:22.134308 12211 raft_consensus.cc:348] T 1ea953f468bb4a1a964dbcb83996c0d0 P 5dd53dc83fe74e8fbfe34f52687c775d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae22c0059ad4d0d8dd22e1cc9b23f49" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35383 } } peers { permanent_uuid: "4d32e51bc30d4fbc8b931d8b792190e9" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38797 } } peers { permanent_uuid: "5dd53dc83fe74e8fbfe34f52687c775d" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 40331 } }
I20260926 20:22:22.134634 12211 raft_consensus.cc:374] T 1ea953f468bb4a1a964dbcb83996c0d0 P 5dd53dc83fe74e8fbfe34f52687c775d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:22:22.134687 12211 raft_consensus.cc:729] T 1ea953f468bb4a1a964dbcb83996c0d0 P 5dd53dc83fe74e8fbfe34f52687c775d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd53dc83fe74e8fbfe34f52687c775d, State: Initialized, Role: FOLLOWER
I20260926 20:22:22.135221 12211 consensus_queue.cc:260] T 1ea953f468bb4a1a964dbcb83996c0d0 P 5dd53dc83fe74e8fbfe34f52687c775d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae22c0059ad4d0d8dd22e1cc9b23f49" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35383 } } peers { permanent_uuid: "4d32e51bc30d4fbc8b931d8b792190e9" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38797 } } peers { permanent_uuid: "5dd53dc83fe74e8fbfe34f52687c775d" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 40331 } }
I20260926 20:22:22.136386 12211 ts_tablet_manager.cc:1428] T 1ea953f468bb4a1a964dbcb83996c0d0 P 5dd53dc83fe74e8fbfe34f52687c775d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 20:22:22.136782 11922 heartbeater.cc:503] Master 127.0.116.254:35699 was elected leader, sending a full tablet report...
I20260926 20:22:22.137295 12212 raft_consensus.cc:348] T 1ea953f468bb4a1a964dbcb83996c0d0 P cae22c0059ad4d0d8dd22e1cc9b23f49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae22c0059ad4d0d8dd22e1cc9b23f49" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35383 } } peers { permanent_uuid: "4d32e51bc30d4fbc8b931d8b792190e9" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38797 } } peers { permanent_uuid: "5dd53dc83fe74e8fbfe34f52687c775d" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 40331 } }
I20260926 20:22:22.137622 12212 raft_consensus.cc:374] T 1ea953f468bb4a1a964dbcb83996c0d0 P cae22c0059ad4d0d8dd22e1cc9b23f49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:22:22.137715 12212 raft_consensus.cc:729] T 1ea953f468bb4a1a964dbcb83996c0d0 P cae22c0059ad4d0d8dd22e1cc9b23f49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae22c0059ad4d0d8dd22e1cc9b23f49, State: Initialized, Role: FOLLOWER
I20260926 20:22:22.138258 12212 consensus_queue.cc:260] T 1ea953f468bb4a1a964dbcb83996c0d0 P cae22c0059ad4d0d8dd22e1cc9b23f49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae22c0059ad4d0d8dd22e1cc9b23f49" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35383 } } peers { permanent_uuid: "4d32e51bc30d4fbc8b931d8b792190e9" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38797 } } peers { permanent_uuid: "5dd53dc83fe74e8fbfe34f52687c775d" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 40331 } }
I20260926 20:22:22.139359 12212 ts_tablet_manager.cc:1428] T 1ea953f468bb4a1a964dbcb83996c0d0 P cae22c0059ad4d0d8dd22e1cc9b23f49: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260926 20:22:22.140321 12193 heartbeater.cc:503] Master 127.0.116.254:35699 was elected leader, sending a full tablet report...
I20260926 20:22:22.141829 12213 tablet_bootstrap.cc:492] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9: No bootstrap required, opened a new log
I20260926 20:22:22.142050 12213 ts_tablet_manager.cc:1397] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.003s
I20260926 20:22:22.147271 12213 raft_consensus.cc:348] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae22c0059ad4d0d8dd22e1cc9b23f49" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35383 } } peers { permanent_uuid: "4d32e51bc30d4fbc8b931d8b792190e9" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38797 } } peers { permanent_uuid: "5dd53dc83fe74e8fbfe34f52687c775d" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 40331 } }
I20260926 20:22:22.147720 12213 raft_consensus.cc:374] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:22:22.147816 12213 raft_consensus.cc:729] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d32e51bc30d4fbc8b931d8b792190e9, State: Initialized, Role: FOLLOWER
I20260926 20:22:22.148329 12213 consensus_queue.cc:260] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae22c0059ad4d0d8dd22e1cc9b23f49" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35383 } } peers { permanent_uuid: "4d32e51bc30d4fbc8b931d8b792190e9" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38797 } } peers { permanent_uuid: "5dd53dc83fe74e8fbfe34f52687c775d" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 40331 } }
I20260926 20:22:22.149250 12059 heartbeater.cc:503] Master 127.0.116.254:35699 was elected leader, sending a full tablet report...
I20260926 20:22:22.149542 12213 ts_tablet_manager.cc:1428] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260926 20:22:22.172818 12219 raft_consensus.cc:482] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:22:22.173079 12219 raft_consensus.cc:504] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae22c0059ad4d0d8dd22e1cc9b23f49" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35383 } } peers { permanent_uuid: "4d32e51bc30d4fbc8b931d8b792190e9" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38797 } } peers { permanent_uuid: "5dd53dc83fe74e8fbfe34f52687c775d" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 40331 } }
I20260926 20:22:22.174587 12219 leader_election.cc:288] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cae22c0059ad4d0d8dd22e1cc9b23f49 (127.0.116.195:35383), 5dd53dc83fe74e8fbfe34f52687c775d (127.0.116.193:40331)
I20260926 20:22:22.180311 12061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:22:22.174766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:35383 (local address 127.0.116.194:37591)
0926 20:22:22.174881 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 20:22:22.174899 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:22:22.174994 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:22:22.176232 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:22:22.176235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:22:22.176250 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:22:22.176364 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:22:22.176371 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.177670 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.177673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:22:22.178812 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:22:22.178824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.179810 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.179814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:22:22.179941 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:22:22.179986 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 20:22:22.180074 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 20:22:22.180159 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":20}
I20260926 20:22:22.180737 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:22:22.175785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:37591 (local address 127.0.116.195:35383)
0926 20:22:22.175895 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 20:22:22.175900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:22:22.175915 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:22:22.175970 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:22:22.175973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:22:22.176033 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:22:22.176107 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:22:22.176503 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.177316 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.178989 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.179676 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.179825 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:22:22.179889 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 20:22:22.180518 (+ 629us) server_negotiation.cc:299] Negotiation successful
0926 20:22:22.180602 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 20:22:22.181460 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea953f468bb4a1a964dbcb83996c0d0" candidate_uuid: "4d32e51bc30d4fbc8b931d8b792190e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae22c0059ad4d0d8dd22e1cc9b23f49" is_pre_election: true
I20260926 20:22:22.181834 12147 raft_consensus.cc:2454] T 1ea953f468bb4a1a964dbcb83996c0d0 P cae22c0059ad4d0d8dd22e1cc9b23f49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d32e51bc30d4fbc8b931d8b792190e9 in term 0.
I20260926 20:22:22.182452 11949 leader_election.cc:302] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d32e51bc30d4fbc8b931d8b792190e9, cae22c0059ad4d0d8dd22e1cc9b23f49; no voters:
I20260926 20:22:22.182809 12219 raft_consensus.cc:2781] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:22:22.182891 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:22:22.175626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:48255 (local address 127.0.116.193:40331)
0926 20:22:22.175744 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 20:22:22.175748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:22:22.176073 (+ 325us) server_negotiation.cc:407] Connection header received
0926 20:22:22.176250 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:22:22.176253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:22:22.176298 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:22:22.176345 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:22:22.176711 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.177423 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.181375 (+ 3952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.182058 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.182170 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:22:22.182230 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 20:22:22.182683 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 20:22:22.182767 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:22:22.182902 12219 raft_consensus.cc:482] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:22:22.182963 12219 raft_consensus.cc:3037] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:22:22.183339 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:22:22.175055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:40331 (local address 127.0.116.194:48255)
0926 20:22:22.175987 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0926 20:22:22.176000 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:22:22.176138 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:22:22.176475 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:22:22.176477 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:22:22.176505 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:22:22.176597 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:22:22.176602 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.177563 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.177566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:22:22.181179 (+ 3613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:22:22.181190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.182432 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.182437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:22:22.182536 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:22:22.182560 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:22:22.183094 (+ 534us) client_negotiation.cc:240] Negotiation successful
0926 20:22:22.183178 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":8,"thread_start_us":110,"threads_started":1}
I20260926 20:22:22.183970 11870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea953f468bb4a1a964dbcb83996c0d0" candidate_uuid: "4d32e51bc30d4fbc8b931d8b792190e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd53dc83fe74e8fbfe34f52687c775d" is_pre_election: true
I20260926 20:22:22.184296 11870 raft_consensus.cc:2454] T 1ea953f468bb4a1a964dbcb83996c0d0 P 5dd53dc83fe74e8fbfe34f52687c775d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d32e51bc30d4fbc8b931d8b792190e9 in term 0.
I20260926 20:22:22.184428 12219 raft_consensus.cc:504] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae22c0059ad4d0d8dd22e1cc9b23f49" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35383 } } peers { permanent_uuid: "4d32e51bc30d4fbc8b931d8b792190e9" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38797 } } peers { permanent_uuid: "5dd53dc83fe74e8fbfe34f52687c775d" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 40331 } }
I20260926 20:22:22.184809 12219 leader_election.cc:288] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9 [CANDIDATE]: Term 1 election: Requested vote from peers cae22c0059ad4d0d8dd22e1cc9b23f49 (127.0.116.195:35383), 5dd53dc83fe74e8fbfe34f52687c775d (127.0.116.193:40331)
I20260926 20:22:22.185596 12147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea953f468bb4a1a964dbcb83996c0d0" candidate_uuid: "4d32e51bc30d4fbc8b931d8b792190e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae22c0059ad4d0d8dd22e1cc9b23f49"
I20260926 20:22:22.185743 12147 raft_consensus.cc:3037] T 1ea953f468bb4a1a964dbcb83996c0d0 P cae22c0059ad4d0d8dd22e1cc9b23f49 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:22:22.187170 12147 raft_consensus.cc:2454] T 1ea953f468bb4a1a964dbcb83996c0d0 P cae22c0059ad4d0d8dd22e1cc9b23f49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d32e51bc30d4fbc8b931d8b792190e9 in term 1.
I20260926 20:22:22.187647 11870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea953f468bb4a1a964dbcb83996c0d0" candidate_uuid: "4d32e51bc30d4fbc8b931d8b792190e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd53dc83fe74e8fbfe34f52687c775d"
I20260926 20:22:22.187785 11870 raft_consensus.cc:3037] T 1ea953f468bb4a1a964dbcb83996c0d0 P 5dd53dc83fe74e8fbfe34f52687c775d [term 0 FOLLOWER]: Advancing to term 1
W20260926 20:22:22.187914 12060 tablet.cc:2369] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:22:22.189165 11870 raft_consensus.cc:2454] T 1ea953f468bb4a1a964dbcb83996c0d0 P 5dd53dc83fe74e8fbfe34f52687c775d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d32e51bc30d4fbc8b931d8b792190e9 in term 1.
I20260926 20:22:22.190181 11949 leader_election.cc:302] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d32e51bc30d4fbc8b931d8b792190e9, 5dd53dc83fe74e8fbfe34f52687c775d; no voters:
I20260926 20:22:22.190599 12219 raft_consensus.cc:2781] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:22:22.190835 12219 raft_consensus.cc:686] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9 [term 1 LEADER]: Becoming Leader. State: Replica: 4d32e51bc30d4fbc8b931d8b792190e9, State: Running, Role: LEADER
I20260926 20:22:22.191236 12219 consensus_queue.cc:237] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae22c0059ad4d0d8dd22e1cc9b23f49" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35383 } } peers { permanent_uuid: "4d32e51bc30d4fbc8b931d8b792190e9" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38797 } } peers { permanent_uuid: "5dd53dc83fe74e8fbfe34f52687c775d" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 40331 } }
I20260926 20:22:22.195773 11709 catalog_manager.cc:5495] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d32e51bc30d4fbc8b931d8b792190e9 (127.0.116.194). New cstate: current_term: 1 leader_uuid: "4d32e51bc30d4fbc8b931d8b792190e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae22c0059ad4d0d8dd22e1cc9b23f49" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d32e51bc30d4fbc8b931d8b792190e9" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5dd53dc83fe74e8fbfe34f52687c775d" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 40331 } health_report { overall_health: UNKNOWN } } }
Sep 26 20:22:22 dist-test-slave-qsm9 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
I20260926 20:22:22.229149 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:22:22.224250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44910 (local address 127.0.116.194:38797)
0926 20:22:22.224367 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 20:22:22.224371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:22:22.224395 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:22:22.224602 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:22:22.224605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:22:22.224674 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:22:22.224783 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:22:22.225162 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.225872 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.227052 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:22:22.227380 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:22:22.227513 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:22:22.227915 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:22:22.228511 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:22:22.228627 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 20:22:22.228946 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 20:22:22.229010 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 20:22:22.237438 12147 raft_consensus.cc:1261] T 1ea953f468bb4a1a964dbcb83996c0d0 P cae22c0059ad4d0d8dd22e1cc9b23f49 [term 1 FOLLOWER]: Refusing update from remote peer 4d32e51bc30d4fbc8b931d8b792190e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:22:22.237483 11870 raft_consensus.cc:1261] T 1ea953f468bb4a1a964dbcb83996c0d0 P 5dd53dc83fe74e8fbfe34f52687c775d [term 1 FOLLOWER]: Refusing update from remote peer 4d32e51bc30d4fbc8b931d8b792190e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:22:22.238215 12219 consensus_queue.cc:1035] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae22c0059ad4d0d8dd22e1cc9b23f49" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:22:22.238400 12222 consensus_queue.cc:1035] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd53dc83fe74e8fbfe34f52687c775d" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 40331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:22:22.255216 12225 mvcc.cc:205] Tried to move back new op lower bound from 7333700166595682304 to 7333700166420963328. Current Snapshot: MvccSnapshot[applied={T|T < 7333700166595682304}]
I20260926 20:22:22.259119 12226 mvcc.cc:205] Tried to move back new op lower bound from 7333700166595682304 to 7333700166420963328. Current Snapshot: MvccSnapshot[applied={T|T < 7333700166595682304}]
I20260926 20:22:22.263648 12227 mvcc.cc:205] Tried to move back new op lower bound from 7333700166595682304 to 7333700166420963328. Current Snapshot: MvccSnapshot[applied={T|T < 7333700166595682304}]
I20260926 20:22:22.275593 11708 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:22:22.275791 11708 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:22:22.285239 11708 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41b9ca6578f94b42a2fbd3ae01a6516c: Sending DeleteTablet for 3 replicas of tablet 1ea953f468bb4a1a964dbcb83996c0d0
I20260926 20:22:22.287721 11985 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea953f468bb4a1a964dbcb83996c0d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:22:22 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:44898
I20260926 20:22:22.287902 11853 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea953f468bb4a1a964dbcb83996c0d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:22:22 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:55444
I20260926 20:22:22.288483 12117 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea953f468bb4a1a964dbcb83996c0d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:22:22 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:58780
I20260926 20:22:22.290586 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 11790
I20260926 20:22:22.291708 12235 tablet_replica.cc:330] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9: stopping tablet replica
I20260926 20:22:22.292099 12235 raft_consensus.cc:2229] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:22:22.292573 12235 raft_consensus.cc:2258] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:22:22.293809 12237 tablet_replica.cc:330] T 1ea953f468bb4a1a964dbcb83996c0d0 P cae22c0059ad4d0d8dd22e1cc9b23f49: stopping tablet replica
I20260926 20:22:22.294198 12237 raft_consensus.cc:2229] T 1ea953f468bb4a1a964dbcb83996c0d0 P cae22c0059ad4d0d8dd22e1cc9b23f49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:22:22.294576 12237 raft_consensus.cc:2258] T 1ea953f468bb4a1a964dbcb83996c0d0 P cae22c0059ad4d0d8dd22e1cc9b23f49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:22:22.295368 12235 ts_tablet_manager.cc:1905] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:22:22.298228 12235 ts_tablet_manager.cc:1918] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:22:22.298336 12235 log.cc:1196] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/1ea953f468bb4a1a964dbcb83996c0d0
I20260926 20:22:22.298637 12235 ts_tablet_manager.cc:1939] T 1ea953f468bb4a1a964dbcb83996c0d0 P 4d32e51bc30d4fbc8b931d8b792190e9: Deleting consensus metadata
W20260926 20:22:22.302340 11696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.116.193:40331 (error 108)
I20260926 20:22:22.303332 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 11929
I20260926 20:22:22.304942 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:22:22.304623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:40331 (local address 127.0.0.1:55456)
0926 20:22:22.304769 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 20:22:22.304847 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.193:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 20:22:22.305146 11696 catalog_manager.cc:4584] TS 5dd53dc83fe74e8fbfe34f52687c775d (127.0.116.193:40331): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ea953f468bb4a1a964dbcb83996c0d0: Network error: Client connection negotiation failed: client connection to 127.0.116.193:40331: connect: Connection refused (error 111)
I20260926 20:22:22.306391 12237 ts_tablet_manager.cc:1905] T 1ea953f468bb4a1a964dbcb83996c0d0 P cae22c0059ad4d0d8dd22e1cc9b23f49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:22:22.309500 12237 ts_tablet_manager.cc:1918] T 1ea953f468bb4a1a964dbcb83996c0d0 P cae22c0059ad4d0d8dd22e1cc9b23f49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:22:22.309623 12237 log.cc:1196] T 1ea953f468bb4a1a964dbcb83996c0d0 P cae22c0059ad4d0d8dd22e1cc9b23f49: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/1ea953f468bb4a1a964dbcb83996c0d0
I20260926 20:22:22.310165 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 12064
I20260926 20:22:22.310191 12237 ts_tablet_manager.cc:1939] T 1ea953f468bb4a1a964dbcb83996c0d0 P cae22c0059ad4d0d8dd22e1cc9b23f49: Deleting consensus metadata
I20260926 20:22:22.317068 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:22:22.316670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:38797 (local address 127.0.0.1:44924)
0926 20:22:22.316890 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 20:22:22.316968 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.194:38797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260926 20:22:22.319020 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 11677
I20260926 20:22:22.325198 467 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 20:22:22.670608 467 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 20:22:22.670785 467 mini_ranger.cc:66] Stopping Ranger...
I20260926 20:22:23.016451 467 mini_ranger.cc:68] Stopped Ranger
2026-09-26 20:22:23.016 UTC [11263] LOG: received smart shutdown request
2026-09-26 20:22:23.020 UTC [11263] LOG: background worker "logical replication launcher" (PID 11272) exited with exit code 1
2026-09-26 20:22:23.020 UTC [11267] LOG: shutting down
2026-09-26 20:22:23.208 UTC [11263] LOG: database system is shut down
2026-09-26T20:22:23Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47208 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:22:23 dist-test-slave-qsm9 krb5kdc[12247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:22:23 dist-test-slave-qsm9 krb5kdc[12247](info): set up 2 sockets
Sep 26 20:22:23 dist-test-slave-qsm9 krb5kdc[12247](info): commencing operation
krb5kdc: starting...
W20260926 20:22:24.368772 467 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:22:24 dist-test-slave-qsm9 krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:22:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:22:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.116.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.116.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.116.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/rangeradmin_127.0.116.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.116.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/rangeradmin_127.0.116.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.116.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.116.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.116.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/rangerlookup_127.0.116.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.116.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/rangerlookup_127.0.116.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.116.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.116.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.116.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/HTTP_127.0.116.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.116.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/HTTP_127.0.116.212@KRBTEST.COM.keytab.
I20260926 20:22:24.546602 467 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task3m9pMa/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/postgres -l logfile start
2026-09-26 20:22:27.331 UTC [12277] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 20:22:27.331 UTC [12277] LOG: listening on IPv4 address "127.0.116.212", port 58833
2026-09-26 20:22:27.335 UTC [12277] LOG: listening on Unix socket "/tmp/.s.PGSQL.58833"
2026-09-26 20:22:27.357 UTC [12280] LOG: database system was shut down at 2026-09-26 20:22:25 UTC
2026-09-26 20:22:27.362 UTC [12277] LOG: database system is ready to accept connections
I20260926 20:22:28.312099 467 mini_postgres.cc:96] Postgres bound to 58833
2026-09-26 20:22:28.318 UTC [12288] FATAL: database "slave" does not exist
127.0.116.212:58833 - accepting connections
I20260926 20:22:28.319240 467 mini_ranger.cc:161] Starting Ranger...
I20260926 20:22:28.334582 467 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 20:22:28.894752 467 mini_ranger.cc:87] Created ranger Postgres database
I20260926 20:22:28.894860 467 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ranger-admin
2026-09-26 20:22:29,166 [I] DB FLAVOR :POSTGRES
2026-09-26 20:22:29,167 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:22:29,167 [I] Checking connection..
2026-09-26 20:22:29,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 20:22:29,438 [I] Checking connection passed.
2026-09-26 20:22:29,438 [I] --------- Verifying version history table ---------
2026-09-26 20:22:29,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:22:29,699 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:22:29,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:22:29,939 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:22:29,939 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 20:22:29,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 20:22:30.169 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 20:22:30,175 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 20:22:30,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:22:30,417 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 20:22:30,417 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:22:30,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 20:22:30,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 20:22:30,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qsm9.c.gcp-upstream.internal','N') ;"
2026-09-26 20:22:31,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:22:31,420 [I] Table x_portal_user does not exist in database ranger
2026-09-26 20:22:31,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:22:31,704 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 20:22:31,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:22:31,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:22:32,194 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 20:22:32,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 20:22:33.262 UTC [12460] WARNING: there is no transaction in progress
2026-09-26 20:22:33.281 UTC [12460] WARNING: there is no transaction in progress
2026-09-26 20:22:33.302 UTC [12460] WARNING: there is no transaction in progress
2026-09-26 20:22:33.321 UTC [12460] WARNING: there is no transaction in progress
2026-09-26 20:22:33.339 UTC [12460] WARNING: there is no transaction in progress
2026-09-26 20:22:33.360 UTC [12460] WARNING: there is no transaction in progress
2026-09-26 20:22:33.428 UTC [12460] WARNING: there is no transaction in progress
2026-09-26 20:22:33.437 UTC [12460] WARNING: there is no transaction in progress
2026-09-26 20:22:33.455 UTC [12460] WARNING: there is no transaction in progress
2026-09-26 20:22:33.467 UTC [12460] WARNING: there is no transaction in progress
2026-09-26 20:22:33.479 UTC [12460] WARNING: there is no transaction in progress
2026-09-26 20:22:33.495 UTC [12460] WARNING: there is no transaction in progress
2026-09-26 20:22:33.507 UTC [12460] WARNING: there is no transaction in progress
2026-09-26 20:22:33.516 UTC [12460] WARNING: there is no transaction in progress
2026-09-26 20:22:33.525 UTC [12460] WARNING: there is no transaction in progress
2026-09-26 20:22:33.534 UTC [12460] WARNING: there is no transaction in progress
2026-09-26 20:22:34,326 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 20:22:34,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 20:22:34,559 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 20:22:34,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:22:34,813 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 20:22:34,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:22:35,056 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 20:22:35,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:22:35,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:22:35,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qsm9.c.gcp-upstream.internal';"
2026-09-26 20:22:35,776 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 20:22:35,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 20:22:36,035 [I] DB_PATCHES have already been applied
I20260926 20:22:36.040706 467 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ranger-admin:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task3m9pMa/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 20:22:36.040774 467 mini_ranger.cc:193] Using host: 127.0.116.212
I20260926 20:22:36.044418 467 mini_ranger.cc:227] Ranger admin URL: http://127.0.116.212:56093
Sep 26, 2026 8:22:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:22:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:22:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 8:22:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 8:22:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 8:22:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 8:22:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 20:22:37.187373 467 mini_ranger.cc:160] Time spent starting Ranger: real 8.868s user 0.000s sys 0.010s
Sep 26, 2026 8:22:56 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 8:22:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 8:22:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 8:22:57 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 8:22:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 8:22:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 8:22:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 8:22:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 8:22:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 8:22:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 8:22:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 8:22:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 8:22:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 8:22:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 8:22:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 8:22:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 8:22:57 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 8:22:57 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 20:22:58.491560 467 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.116.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.116.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.116.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/rangerkms_127.0.116.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.116.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/rangerkms_127.0.116.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.116.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.116.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.116.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/HTTP_127.0.116.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.116.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/HTTP_127.0.116.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 20:22:58 dist-test-slave-qsm9 krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454178, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 20:22:58.583995 467 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 20:22:58.593744 467 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 20:22:59.556434 467 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 20:22:59.556509 467 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ranger-kms
I20260926 20:22:59.556522 467 mini_ranger_kms.cc:227] Using postgres at 127.0.116.212:58833
2026-09-26 20:22:59,741 [I] DB FLAVOR :POSTGRES
2026-09-26 20:22:59,742 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:22:59,742 [I] Checking connection
2026-09-26 20:22:59,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 20:22:59,999 [I] connection success
2026-09-26 20:22:59,999 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 20:22:59,999 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 20:23:00,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 20:23:00,250 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 20:23:00,250 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:23:00,251 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 20:23:00,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.116.212:58833/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 20:23:00,525 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 20:23:01.037149 467 mini_ranger_kms.cc:310] Created kms service
I20260926 20:23:01.258813 467 mini_ranger_kms.cc:326] Created kudu user
I20260926 20:23:01.304478 467 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 20:23:01 dist-test-slave-qsm9 krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454178, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
I20260926 20:23:01.729624 467 mini_ranger_kms.cc:384] Added ranger policy
I20260926 20:23:01.729784 467 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ranger-kms:/tmp/dist-test-task3m9pMa/build/debug/bin/postgresql.jar:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task3m9pMa/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 20:23:01.729806 467 mini_ranger_kms.cc:240] Using host: 127.0.116.212
I20260926 20:23:01.734067 467 mini_ranger_kms.cc:276] Ranger KMS PID: 12641
I20260926 20:23:01.734180 467 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.116.212:40019
Sep 26, 2026 8:23:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:23:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:23:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 8:23:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 8:23:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 8:23:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 20:23:04.144874 467 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.561s user 0.013s sys 0.154s
I20260926 20:23:04.145013 467 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 20:23:04.145099 467 mini_ranger_kms.cc:401] 127.0.116.212:40019/kms/v1/keys
Sep 26 20:23:04 dist-test-slave-qsm9 krb5kdc[12247](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790454184, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.116.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 20:23:04 dist-test-slave-qsm9 krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454184, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.116.212@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
Sep 26 20:23:05 dist-test-slave-qsm9 krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454184, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.116.212@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
Sep 26 20:23:07 dist-test-slave-qsm9 krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:23:07.691663 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:44213
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:44213
--ranger_config_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.116.212:40019/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:23:07.827745 12692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:07.828033 12692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:07.828073 12692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:07.832280 12692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:23:07.832352 12692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:07.832372 12692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:07.832389 12692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:23:07.832405 12692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:23:07.838256 12692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.116.212:40019/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:44213
--ranger_config_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:44213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:07.839403 12692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:07.840570 12692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:07.849704 12698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:07.849828 12692 server_base.cc:1034] running on GCE node
W20260926 20:23:07.849706 12700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:07.849717 12697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:07.850581 12692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:07.851848 12692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:07.853050 12692 hybrid_clock.cc:648] HybridClock initialized: now 1790454187853041 us; error 32 us; skew 500 ppm
Sep 26 20:23:07 dist-test-slave-qsm9 krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:07.856122 12692 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:23:07.857116 12692 webserver.cc:460] Webserver started at http://127.0.116.254:40001/ using document root <none> and password file <none>
I20260926 20:23:07.857707 12692 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:07.857800 12692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:07.858026 12692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:23:07 dist-test-slave-qsm9 krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
I20260926 20:23:07.908926 12692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "39d341d5c84248d282e716ce2544f884"
format_stamp: "Formatted at 2026-09-26 20:23:07 on dist-test-slave-qsm9"
server_key: "783081e6dd4d63b12c83cec7b91359d1"
server_key_iv: "89d0a7413c9c96c98092cbe1d215a0d5"
server_key_version: "kuduclusterkey@0"
I20260926 20:23:07.909636 12692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "39d341d5c84248d282e716ce2544f884"
format_stamp: "Formatted at 2026-09-26 20:23:07 on dist-test-slave-qsm9"
server_key: "783081e6dd4d63b12c83cec7b91359d1"
server_key_iv: "89d0a7413c9c96c98092cbe1d215a0d5"
server_key_version: "kuduclusterkey@0"
I20260926 20:23:07.913982 12692 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:23:07.934439 12707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:07.936056 12692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:23:07.936177 12692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "39d341d5c84248d282e716ce2544f884"
format_stamp: "Formatted at 2026-09-26 20:23:07 on dist-test-slave-qsm9"
server_key: "783081e6dd4d63b12c83cec7b91359d1"
server_key_iv: "89d0a7413c9c96c98092cbe1d215a0d5"
server_key_version: "kuduclusterkey@0"
I20260926 20:23:07.936358 12692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:07.949380 12692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:07.952543 12692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:07.952795 12692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:07.960341 12692 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:44213
I20260926 20:23:07.960366 12759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:44213 every 8 connection(s)
I20260926 20:23:07.961388 12692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:23:07.964594 12760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:23:07.970619 12760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884: Bootstrap starting.
I20260926 20:23:07.971026 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 12692
I20260926 20:23:07.971187 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:23:07.971544 467 external_mini_cluster.cc:1428] Setting key 521aabccf767499b06a9e4ed933973fb
I20260926 20:23:07.973512 12760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884: Neither blocks nor log segments found. Creating new log.
I20260926 20:23:07.974263 12760 log.cc:824] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884: Log is configured to *not* fsync() on all Append() calls
I20260926 20:23:07.976514 12760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884: No bootstrap required, opened a new log
Sep 26 20:23:07 dist-test-slave-qsm9 krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:23:07.981343 12760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d341d5c84248d282e716ce2544f884" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44213 } }
I20260926 20:23:07.981596 12760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:23:07.981698 12760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d341d5c84248d282e716ce2544f884, State: Initialized, Role: FOLLOWER
I20260926 20:23:07.982261 12760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d341d5c84248d282e716ce2544f884" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44213 } }
I20260926 20:23:07.982455 12760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:23:07.982537 12760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:23:07.982661 12760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:23:07.983933 12760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d341d5c84248d282e716ce2544f884" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44213 } }
I20260926 20:23:07.984311 12760 leader_election.cc:302] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39d341d5c84248d282e716ce2544f884; no voters:
I20260926 20:23:07.984647 12760 leader_election.cc:288] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:23:07.984783 12765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:23:07.985158 12765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [term 1 LEADER]: Becoming Leader. State: Replica: 39d341d5c84248d282e716ce2544f884, State: Running, Role: LEADER
I20260926 20:23:07.985532 12765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d341d5c84248d282e716ce2544f884" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44213 } }
I20260926 20:23:07.986055 12760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:23:07.986792 12765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39d341d5c84248d282e716ce2544f884. Latest consensus state: current_term: 1 leader_uuid: "39d341d5c84248d282e716ce2544f884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d341d5c84248d282e716ce2544f884" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44213 } } }
I20260926 20:23:07.986907 12765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [sys.catalog]: This master's current role is: LEADER
I20260926 20:23:07.987650 12772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:23:07.987841 12763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:07.972973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.0.116.254:44213)
0926 20:23:07.973430 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:07.973445 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:07.973479 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:23:07.974188 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:07.974206 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:07.974594 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:07.974996 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:07.975958 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:07.976996 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:07.978249 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:07.978736 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:07.978861 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:07.981821 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:07.981828 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:07.981846 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:07.981877 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:07.983944 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:07.984546 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:07.984551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:07.984556 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:07.984791 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:07.985136 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:07.985139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:07.985141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:07.985646 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:07.985837 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:07.986306 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 20:23:07.986647 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":151,"threads_started":1}
I20260926 20:23:07.989362 12766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39d341d5c84248d282e716ce2544f884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d341d5c84248d282e716ce2544f884" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44213 } } }
I20260926 20:23:07.989481 12766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [sys.catalog]: This master's current role is: LEADER
I20260926 20:23:07.991400 12772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:23:07.991870 12760 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 20:23:08.001475 12772 catalog_manager.cc:1347] Generated new cluster ID: fa15cb6260d94b92b4652de4447cadf2
I20260926 20:23:08.001595 12772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:23:08.011771 12772 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:23:08.012867 12772 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:23:08.024804 12772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884: Generated new TSK 0
I20260926 20:23:08.025637 12772 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
I20260926 20:23:08.727408 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.116.212:40019/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44213
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:23:08.849949 12805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:08.850279 12805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:08.850395 12805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:08.854888 12805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:08.854986 12805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:08.855083 12805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:23:08.861004 12805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.116.212:40019/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:08.862385 12805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:08.863569 12805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:08.876601 12810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:08.876569 12811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:08.876602 12813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:08.877055 12805 server_base.cc:1034] running on GCE node
I20260926 20:23:08.877595 12805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:08.878486 12805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:08.879714 12805 hybrid_clock.cc:648] HybridClock initialized: now 1790454188879686 us; error 41 us; skew 500 ppm
Sep 26 20:23:08 dist-test-slave-qsm9 krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:08.884339 12805 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:23:08.885423 12805 webserver.cc:460] Webserver started at http://127.0.116.193:36979/ using document root <none> and password file <none>
I20260926 20:23:08.886061 12805 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:08.886179 12805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:08.886559 12805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:23:08 dist-test-slave-qsm9 krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
I20260926 20:23:08.901060 12805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "3981288b43eb4f7d82168b892110eeef"
format_stamp: "Formatted at 2026-09-26 20:23:08 on dist-test-slave-qsm9"
server_key: "c58f5fa0aed54c98233491825181913e"
server_key_iv: "32cfc1f2806017cc06a895e534a9c38a"
server_key_version: "kuduclusterkey@0"
I20260926 20:23:08.901952 12805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "3981288b43eb4f7d82168b892110eeef"
format_stamp: "Formatted at 2026-09-26 20:23:08 on dist-test-slave-qsm9"
server_key: "c58f5fa0aed54c98233491825181913e"
server_key_iv: "32cfc1f2806017cc06a895e534a9c38a"
server_key_version: "kuduclusterkey@0"
I20260926 20:23:08.907795 12805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:23:08.930169 12822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:08.931908 12805 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260926 20:23:08.932070 12805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "3981288b43eb4f7d82168b892110eeef"
format_stamp: "Formatted at 2026-09-26 20:23:08 on dist-test-slave-qsm9"
server_key: "c58f5fa0aed54c98233491825181913e"
server_key_iv: "32cfc1f2806017cc06a895e534a9c38a"
server_key_version: "kuduclusterkey@0"
I20260926 20:23:08.932227 12805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:08.980875 12805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:08.984882 12805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:08.985109 12805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:08.986253 12805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:23:08.987419 12805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:23:08.987565 12805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:08.987718 12805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:23:08.987851 12805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:09.001930 12805 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:46333
I20260926 20:23:09.003243 12805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:23:09.007897 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 12805
I20260926 20:23:09.008030 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:23:09.008442 467 external_mini_cluster.cc:1428] Setting key efa5758a84ff66b2091ebba87babbb14
I20260926 20:23:09.004868 12937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:46333 every 8 connection(s)
Sep 26 20:23:09 dist-test-slave-qsm9 krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.116.194@KRBTEST.COM; defaulting to no policy
I20260926 20:23:09.026587 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:09.008946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:59887 (local address 127.0.116.254:44213)
0926 20:23:09.012099 (+ 3153us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:09.012108 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:09.012124 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:23:09.012179 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:09.012182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:09.012232 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:09.012328 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:09.013999 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:09.014900 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:09.016843 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:09.017226 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:09.017309 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:09.021387 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:09.021391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:09.021394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:09.021432 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:09.023593 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:09.024451 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:09.024457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:09.024459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:09.024578 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:09.025048 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:09.025057 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:09.025061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:09.025317 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:09.025405 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:09.026232 (+ 827us) server_negotiation.cc:299] Negotiation successful
0926 20:23:09.026381 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3056,"thread_start_us":133,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.194@KRBTEST.COM" created.
I20260926 20:23:09.028685 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:09.007634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44213 (local address 127.0.116.193:59887)
0926 20:23:09.009888 (+ 2254us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:09.009956 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:09.011054 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:09.012675 (+ 1621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:09.012685 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:09.013081 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:09.013805 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:09.013825 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:09.015078 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:09.015081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:09.016673 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:09.016688 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:09.017549 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:09.017553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:09.017783 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:09.019064 (+ 1281us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:09.019087 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:09.021052 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:09.023834 (+ 2782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:09.023841 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:09.023862 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:09.024303 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:09.024705 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:09.024708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:09.024711 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:09.024929 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:09.025475 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:09.025482 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:09.025871 (+ 389us) client_negotiation.cc:769] Sending connection context
0926 20:23:09.026863 (+ 992us) client_negotiation.cc:240] Negotiation successful
0926 20:23:09.027286 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1932,"thread_start_us":146,"threads_started":1}
I20260926 20:23:09.031368 12938 heartbeater.cc:347] Connected to a master server at 127.0.116.254:44213
I20260926 20:23:09.031710 12938 heartbeater.cc:464] Registering TS with master...
I20260926 20:23:09.032392 12938 heartbeater.cc:511] Master 127.0.116.254:44213 requested a full tablet report, sending...
I20260926 20:23:09.034409 12724 ts_manager.cc:194] Registered new tserver with Master: 3981288b43eb4f7d82168b892110eeef (127.0.116.193:46333)
I20260926 20:23:09.036042 12724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:59887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
I20260926 20:23:09.051048 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.116.212:40019/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44213
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:23:09 dist-test-slave-qsm9 krb5kdc[12247](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790454189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 20:23:09.201777 12944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:09.202214 12944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:09.202284 12944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:09.209117 12944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:09.209299 12944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:09.209421 12944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:23:09.218221 12944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.116.212:40019/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:09.219667 12944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:09.221033 12944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:09.233552 12951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:09.233670 12953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:09.233552 12950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:09.234841 12944 server_base.cc:1034] running on GCE node
I20260926 20:23:09.235388 12944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:09.236253 12944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:09.237452 12944 hybrid_clock.cc:648] HybridClock initialized: now 1790454189237428 us; error 43 us; skew 500 ppm
Sep 26 20:23:09 dist-test-slave-qsm9 krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:09.241343 12944 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:23:09.242564 12944 webserver.cc:460] Webserver started at http://127.0.116.194:43867/ using document root <none> and password file <none>
I20260926 20:23:09.243307 12944 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:09.243396 12944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:09.243676 12944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:23:09 dist-test-slave-qsm9 krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
I20260926 20:23:09.260455 12944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "da1339f6b1ae4efd859146c265b90ffb"
format_stamp: "Formatted at 2026-09-26 20:23:09 on dist-test-slave-qsm9"
server_key: "0bd56ad46cccd9db732d3998a137d711"
server_key_iv: "3607ef8189b04638699e657cb5c3e8b3"
server_key_version: "kuduclusterkey@0"
I20260926 20:23:09.261346 12944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "da1339f6b1ae4efd859146c265b90ffb"
format_stamp: "Formatted at 2026-09-26 20:23:09 on dist-test-slave-qsm9"
server_key: "0bd56ad46cccd9db732d3998a137d711"
server_key_iv: "3607ef8189b04638699e657cb5c3e8b3"
server_key_version: "kuduclusterkey@0"
I20260926 20:23:09.269189 12944 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260926 20:23:09.299604 12960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:09.301963 12944 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.003s sys 0.000s
I20260926 20:23:09.302220 12944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "da1339f6b1ae4efd859146c265b90ffb"
format_stamp: "Formatted at 2026-09-26 20:23:09 on dist-test-slave-qsm9"
server_key: "0bd56ad46cccd9db732d3998a137d711"
server_key_iv: "3607ef8189b04638699e657cb5c3e8b3"
server_key_version: "kuduclusterkey@0"
I20260926 20:23:09.302446 12944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:09.343423 12944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:09.348544 12944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:09.348820 12944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:09.354157 12944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:23:09.355542 12944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:23:09.355621 12944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:09.355686 12944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:23:09.355762 12944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:09.366855 12944 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:36859
I20260926 20:23:09.368256 12944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:23:09.369438 13073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:36859 every 8 connection(s)
I20260926 20:23:09.372308 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 12944
I20260926 20:23:09.372438 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:23:09.372836 467 external_mini_cluster.cc:1428] Setting key 21ff40fe46e6f3f1590713b28b1dfd3b
Sep 26 20:23:09 dist-test-slave-qsm9 krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.116.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.195@KRBTEST.COM" created.
I20260926 20:23:09.392958 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:09.371646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:33157 (local address 127.0.116.254:44213)
0926 20:23:09.371825 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:09.371830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:09.376004 (+ 4174us) server_negotiation.cc:407] Connection header received
0926 20:23:09.377251 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:09.377254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:09.377311 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:09.377422 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:09.378875 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:09.379791 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:09.381523 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:09.381894 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:09.382019 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:09.388097 (+ 6078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:09.388112 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:09.388115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:09.388149 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:09.390024 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:09.390888 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:09.390892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:09.390894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:09.390998 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:09.391463 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:09.391467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:09.391469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:09.391866 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:09.391978 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:09.392635 (+ 657us) server_negotiation.cc:299] Negotiation successful
0926 20:23:09.392779 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 20:23:09.395114 13076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:09.372133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44213 (local address 127.0.116.194:33157)
0926 20:23:09.375846 (+ 3713us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:09.375909 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:09.376987 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:09.377645 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:09.377655 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:09.378022 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:09.378700 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:09.378719 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:09.379949 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:09.379952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:09.381379 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:09.381390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:09.382089 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:09.382093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:09.382303 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:09.383048 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:09.383069 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:09.384934 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:09.390165 (+ 5231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:09.390173 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:09.390193 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:09.390588 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:09.391120 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:09.391123 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:09.391125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:09.391339 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:09.391974 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:09.391981 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:09.392343 (+ 362us) client_negotiation.cc:769] Sending connection context
0926 20:23:09.393487 (+ 1144us) client_negotiation.cc:240] Negotiation successful
0926 20:23:09.393902 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3467,"thread_start_us":138,"threads_started":1}
I20260926 20:23:09.397064 13074 heartbeater.cc:347] Connected to a master server at 127.0.116.254:44213
I20260926 20:23:09.397333 13074 heartbeater.cc:464] Registering TS with master...
I20260926 20:23:09.397922 13074 heartbeater.cc:511] Master 127.0.116.254:44213 requested a full tablet report, sending...
I20260926 20:23:09.399163 12724 ts_manager.cc:194] Registered new tserver with Master: da1339f6b1ae4efd859146c265b90ffb (127.0.116.194:36859)
I20260926 20:23:09.400007 12724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:33157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
I20260926 20:23:09.419544 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.116.212:40019/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44213
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:23:09 dist-test-slave-qsm9 krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
W20260926 20:23:09.581032 13079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:09.581451 13079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:09.581543 13079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:09.587617 13079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:09.587781 13079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:09.587921 13079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:23:09.595990 13079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.116.212:40019/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:09.597674 13079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:09.599121 13079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:09.613765 13085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:09.613770 13087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:09.614367 13084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:09.615221 13079 server_base.cc:1034] running on GCE node
I20260926 20:23:09.615736 13079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:09.616534 13079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:09.632088 13079 hybrid_clock.cc:648] HybridClock initialized: now 1790454189631986 us; error 131 us; skew 500 ppm
Sep 26 20:23:09 dist-test-slave-qsm9 krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:09.635363 13079 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:23:09.636590 13079 webserver.cc:460] Webserver started at http://127.0.116.195:38947/ using document root <none> and password file <none>
I20260926 20:23:09.637315 13079 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:09.637426 13079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:09.637669 13079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:23:09 dist-test-slave-qsm9 krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
I20260926 20:23:09.651135 13079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "3ab62b6703d84f7b82dcf54023c46c83"
format_stamp: "Formatted at 2026-09-26 20:23:09 on dist-test-slave-qsm9"
server_key: "968d8617d30ca5d969a9dfd71283542f"
server_key_iv: "157ec7564605026f472fae7595611654"
server_key_version: "kuduclusterkey@0"
I20260926 20:23:09.652072 13079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "3ab62b6703d84f7b82dcf54023c46c83"
format_stamp: "Formatted at 2026-09-26 20:23:09 on dist-test-slave-qsm9"
server_key: "968d8617d30ca5d969a9dfd71283542f"
server_key_iv: "157ec7564605026f472fae7595611654"
server_key_version: "kuduclusterkey@0"
I20260926 20:23:09.671974 13079 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.004s sys 0.000s
I20260926 20:23:09.689031 13094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:09.690989 13079 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:23:09.691128 13079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "3ab62b6703d84f7b82dcf54023c46c83"
format_stamp: "Formatted at 2026-09-26 20:23:09 on dist-test-slave-qsm9"
server_key: "968d8617d30ca5d969a9dfd71283542f"
server_key_iv: "157ec7564605026f472fae7595611654"
server_key_version: "kuduclusterkey@0"
I20260926 20:23:09.691267 13079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:09.713743 13079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:09.717684 13079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:09.717895 13079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:09.718590 13079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:23:09.719662 13079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:23:09.719777 13079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:09.719868 13079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:23:09.719918 13079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:09.730392 13079 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:45625
I20260926 20:23:09.731674 13079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:23:09.732015 13207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:45625 every 8 connection(s)
I20260926 20:23:09.740360 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 13079
I20260926 20:23:09.740497 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:23:09.740823 467 external_mini_cluster.cc:1428] Setting key bca7ac3df9268ff34383f5fd38a97e05
Sep 26 20:23:09 dist-test-slave-qsm9 krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:23:09.749897 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:09.734309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:34153 (local address 127.0.116.254:44213)
0926 20:23:09.734440 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:09.734444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:09.735698 (+ 1254us) server_negotiation.cc:407] Connection header received
0926 20:23:09.736996 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:09.736999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:09.737057 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:09.737163 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:09.738556 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:09.739398 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:09.741602 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:09.741972 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:09.742061 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:09.745360 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:09.745364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:09.745367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:09.745399 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:09.747242 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:09.748014 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:09.748018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:09.748020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:09.748126 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:09.748571 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:09.748574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:09.748576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:09.748813 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:09.748907 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:09.749591 (+ 684us) server_negotiation.cc:299] Negotiation successful
0926 20:23:09.749729 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:23:09.751055 13210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:09.734861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44213 (local address 127.0.116.195:34153)
0926 20:23:09.735436 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:09.735524 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:09.736734 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:09.737338 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:09.737348 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:09.737726 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:09.738382 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:09.738400 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:09.739694 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:09.739697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:09.741249 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:09.741262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:09.742288 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:09.742293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:09.742508 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:09.743269 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:09.743294 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:09.745141 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:09.747398 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:09.747405 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:09.747426 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:09.747877 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:09.748249 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:09.748252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:09.748254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:09.748458 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:09.748928 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:09.748936 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:09.749299 (+ 363us) client_negotiation.cc:769] Sending connection context
0926 20:23:09.749515 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 20:23:09.749909 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":157,"threads_started":1}
I20260926 20:23:09.752353 13208 heartbeater.cc:347] Connected to a master server at 127.0.116.254:44213
I20260926 20:23:09.752631 13208 heartbeater.cc:464] Registering TS with master...
I20260926 20:23:09.753232 13208 heartbeater.cc:511] Master 127.0.116.254:44213 requested a full tablet report, sending...
I20260926 20:23:09.754410 12724 ts_manager.cc:194] Registered new tserver with Master: 3ab62b6703d84f7b82dcf54023c46c83 (127.0.116.195:45625)
I20260926 20:23:09.755172 12724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:34153
I20260926 20:23:09.755393 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:23:09.756151 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 12805
I20260926 20:23:09.763581 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 12944
I20260926 20:23:09.771880 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 13079
I20260926 20:23:09.779861 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 12692
I20260926 20:23:09.785172 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:44213
--webserver_interface=127.0.116.254
--webserver_port=40001
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:44213
--ranger_config_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.116.212:40019/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.116.212:40019/kms with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:23:09.899001 13211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:09.899298 13211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:09.899339 13211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:09.903697 13211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:23:09.903759 13211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:09.903779 13211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:09.903795 13211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:23:09.903810 13211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:23:09.909443 13211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.116.212:40019/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:44213
--ranger_config_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:44213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=40001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:09.910526 13211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:09.911639 13211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:09.920251 13219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:09.920241 13217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:09.920235 13216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:09.920297 13211 server_base.cc:1034] running on GCE node
I20260926 20:23:09.921105 13211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:09.922315 13211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:09.923568 13211 hybrid_clock.cc:648] HybridClock initialized: now 1790454189923529 us; error 56 us; skew 500 ppm
Sep 26 20:23:09 dist-test-slave-qsm9 krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:09.926308 13211 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:23:09.927256 13211 webserver.cc:460] Webserver started at http://127.0.116.254:40001/ using document root <none> and password file <none>
I20260926 20:23:09.927906 13211 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:09.928007 13211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:23:09 dist-test-slave-qsm9 krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
I20260926 20:23:09.994771 13211 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:23:09.997691 13227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:09.999266 13211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:23:09.999449 13211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "39d341d5c84248d282e716ce2544f884"
format_stamp: "Formatted at 2026-09-26 20:23:07 on dist-test-slave-qsm9"
server_key: "783081e6dd4d63b12c83cec7b91359d1"
server_key_iv: "89d0a7413c9c96c98092cbe1d215a0d5"
server_key_version: "kuduclusterkey@0"
I20260926 20:23:10.000204 13211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:10.018078 13211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:10.021752 13211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:10.022020 13211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:10.030406 13211 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:44213
I20260926 20:23:10.030440 13279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:44213 every 8 connection(s)
I20260926 20:23:10.031646 13211 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:23:10.033651 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 13211
I20260926 20:23:10.034181 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.116.212:40019/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:46333
--local_ip_for_outbound_sockets=127.0.116.193
--tserver_master_addrs=127.0.116.254:44213
--webserver_port=36979
--webserver_interface=127.0.116.193
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.116.212:40019/kms with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:23:10.036966 13280 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:23:10.038767 13280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884: Bootstrap starting.
I20260926 20:23:10.055985 13280 log.cc:824] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884: Log is configured to *not* fsync() on all Append() calls
I20260926 20:23:10.063333 13280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:23:10.063833 13280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884: Bootstrap complete.
I20260926 20:23:10.068329 13280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d341d5c84248d282e716ce2544f884" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44213 } }
I20260926 20:23:10.068800 13280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d341d5c84248d282e716ce2544f884, State: Initialized, Role: FOLLOWER
I20260926 20:23:10.069327 13280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d341d5c84248d282e716ce2544f884" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44213 } }
I20260926 20:23:10.069440 13280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:23:10.069532 13280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:23:10.069640 13280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:23:10.071707 13280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d341d5c84248d282e716ce2544f884" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44213 } }
I20260926 20:23:10.072059 13280 leader_election.cc:302] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39d341d5c84248d282e716ce2544f884; no voters:
I20260926 20:23:10.072419 13280 leader_election.cc:288] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:23:10.072510 13284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:23:10.072767 13284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [term 2 LEADER]: Becoming Leader. State: Replica: 39d341d5c84248d282e716ce2544f884, State: Running, Role: LEADER
I20260926 20:23:10.073143 13284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d341d5c84248d282e716ce2544f884" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44213 } }
I20260926 20:23:10.073752 13280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:23:10.074548 13286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39d341d5c84248d282e716ce2544f884. Latest consensus state: current_term: 2 leader_uuid: "39d341d5c84248d282e716ce2544f884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d341d5c84248d282e716ce2544f884" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44213 } } }
I20260926 20:23:10.074682 13286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [sys.catalog]: This master's current role is: LEADER
I20260926 20:23:10.074714 13285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "39d341d5c84248d282e716ce2544f884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d341d5c84248d282e716ce2544f884" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44213 } } }
I20260926 20:23:10.074824 13285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884 [sys.catalog]: This master's current role is: LEADER
I20260926 20:23:10.076246 13280 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 20:23:10.076522 13293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:23:10.083244 13293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:23:10.084404 13293 catalog_manager.cc:1259] Loaded cluster ID: fa15cb6260d94b92b4652de4447cadf2
I20260926 20:23:10.084540 13293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:23:10.087702 13293 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:23:10.088689 13293 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884: Loaded TSK: 0
I20260926 20:23:10.090312 13293 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:23:10.191661 13282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:10.192149 13282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:10.192273 13282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:10.198535 13282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:10.198627 13282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:10.198719 13282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:23:10.206720 13282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.116.212:40019/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:46333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=36979
--tserver_master_addrs=127.0.116.254:44213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:10.208359 13282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:10.209681 13282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:10.222304 13317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:10.222555 13318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:10.222745 13320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:10.223610 13282 server_base.cc:1034] running on GCE node
I20260926 20:23:10.224079 13282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:10.224890 13282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:10.226097 13282 hybrid_clock.cc:648] HybridClock initialized: now 1790454190226075 us; error 37 us; skew 500 ppm
Sep 26 20:23:10 dist-test-slave-qsm9 krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:10.230023 13282 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:23:10.231204 13282 webserver.cc:460] Webserver started at http://127.0.116.193:36979/ using document root <none> and password file <none>
I20260926 20:23:10.232062 13282 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:10.232189 13282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:23:10 dist-test-slave-qsm9 krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
I20260926 20:23:10.280627 13282 fs_manager.cc:709] Time spent opening directory manager: real 0.026s user 0.003s sys 0.000s
I20260926 20:23:10.287897 13328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:10.289712 13282 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260926 20:23:10.289882 13282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "3981288b43eb4f7d82168b892110eeef"
format_stamp: "Formatted at 2026-09-26 20:23:08 on dist-test-slave-qsm9"
server_key: "c58f5fa0aed54c98233491825181913e"
server_key_iv: "32cfc1f2806017cc06a895e534a9c38a"
server_key_version: "kuduclusterkey@0"
I20260926 20:23:10.290503 13282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:10.311710 13282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:10.315547 13282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:10.315833 13282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:10.316708 13282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:23:10.317698 13282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:23:10.317777 13282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:10.317850 13282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:23:10.317878 13282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:10.329849 13441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:46333 every 8 connection(s)
I20260926 20:23:10.329849 13282 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:46333
I20260926 20:23:10.331421 13282 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:23:10.333820 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 13282
I20260926 20:23:10.334357 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.116.212:40019/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:36859
--local_ip_for_outbound_sockets=127.0.116.194
--tserver_master_addrs=127.0.116.254:44213
--webserver_port=43867
--webserver_interface=127.0.116.194
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.116.212:40019/kms with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:23:10 dist-test-slave-qsm9 krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:23:10.363709 13444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:10.334254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44213 (local address 127.0.116.193:42115)
0926 20:23:10.341425 (+ 7171us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:10.341509 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:10.342724 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:10.345079 (+ 2355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:10.345089 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:10.345472 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:10.346188 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:10.346211 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:10.348221 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:10.348224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:10.349658 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:10.349669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:10.351035 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:10.351040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:10.351250 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:10.352008 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:10.352031 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:10.353668 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:10.359315 (+ 5647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:10.359325 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:10.359343 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:10.359855 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:10.360397 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:10.360400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:10.360402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:10.360615 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:10.361577 (+ 962us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:10.361584 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:10.361943 (+ 359us) client_negotiation.cc:769] Sending connection context
0926 20:23:10.362145 (+ 202us) client_negotiation.cc:240] Negotiation successful
0926 20:23:10.362531 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6874,"thread_start_us":165,"threads_started":1}
I20260926 20:23:10.365633 13446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:10.335892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:42115 (local address 127.0.116.254:44213)
0926 20:23:10.336399 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:10.336422 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:10.343141 (+ 6719us) server_negotiation.cc:407] Connection header received
0926 20:23:10.343544 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:10.343566 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:10.344083 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:10.344487 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:10.346413 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:10.348045 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:10.349824 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:10.350620 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:10.350797 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:10.353956 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:10.353964 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:10.353985 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:10.354022 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:10.359099 (+ 5077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:10.360032 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:10.360040 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:10.360047 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:10.360244 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:10.360746 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:10.360749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:10.360751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:10.361353 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:10.361497 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:10.364309 (+ 2812us) server_negotiation.cc:299] Negotiation successful
0926 20:23:10.364647 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":120,"threads_started":1}
I20260926 20:23:10.369264 13442 heartbeater.cc:347] Connected to a master server at 127.0.116.254:44213
I20260926 20:23:10.369513 13442 heartbeater.cc:464] Registering TS with master...
I20260926 20:23:10.370105 13442 heartbeater.cc:511] Master 127.0.116.254:44213 requested a full tablet report, sending...
W20260926 20:23:10.372529 13442 heartbeater.cc:644] Failed to heartbeat to 127.0.116.254:44213 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 20:23:10.372664 13442 heartbeater.cc:464] Registering TS with master...
I20260926 20:23:10.372857 13442 heartbeater.cc:511] Master 127.0.116.254:44213 requested a full tablet report, sending...
I20260926 20:23:10.373821 13442 heartbeater.cc:464] Registering TS with master...
I20260926 20:23:10.374007 13442 heartbeater.cc:511] Master 127.0.116.254:44213 requested a full tablet report, sending...
W20260926 20:23:10.374818 13442 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:23:10.483191 13445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:10.483794 13445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:10.483944 13445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:10.490423 13445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:10.490590 13445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:10.490725 13445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:23:10.499234 13445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.116.212:40019/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:36859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=43867
--tserver_master_addrs=127.0.116.254:44213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:10.500939 13445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:10.502379 13445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:10.513783 13451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:10.513782 13454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:10.513787 13452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:10.515771 13445 server_base.cc:1034] running on GCE node
I20260926 20:23:10.516245 13445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:10.517028 13445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:10.518209 13445 hybrid_clock.cc:648] HybridClock initialized: now 1790454190518178 us; error 50 us; skew 500 ppm
Sep 26 20:23:10 dist-test-slave-qsm9 krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:10.521301 13445 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:23:10.522506 13445 webserver.cc:460] Webserver started at http://127.0.116.194:43867/ using document root <none> and password file <none>
I20260926 20:23:10.523089 13445 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:10.523206 13445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:23:10 dist-test-slave-qsm9 krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
I20260926 20:23:10.553412 13445 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.003s sys 0.000s
I20260926 20:23:10.560379 13462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:10.562700 13445 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.001s
I20260926 20:23:10.562893 13445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "da1339f6b1ae4efd859146c265b90ffb"
format_stamp: "Formatted at 2026-09-26 20:23:09 on dist-test-slave-qsm9"
server_key: "0bd56ad46cccd9db732d3998a137d711"
server_key_iv: "3607ef8189b04638699e657cb5c3e8b3"
server_key_version: "kuduclusterkey@0"
I20260926 20:23:10.563875 13445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:10.582717 13445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:10.586767 13445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:10.587008 13445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:10.587807 13445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:23:10.588881 13445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:23:10.588943 13445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:10.589001 13445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:23:10.589023 13445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:10.601238 13445 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:36859
I20260926 20:23:10.601333 13577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:36859 every 8 connection(s)
I20260926 20:23:10.602519 13445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:23:10.604692 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 13445
I20260926 20:23:10.605118 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.116.212:40019/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:45625
--local_ip_for_outbound_sockets=127.0.116.195
--tserver_master_addrs=127.0.116.254:44213
--webserver_port=38947
--webserver_interface=127.0.116.195
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.116.212:40019/kms with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:23:10 dist-test-slave-qsm9 krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:23:10.634335 13446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:10.610565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:58099 (local address 127.0.116.254:44213)
0926 20:23:10.610711 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:10.610715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:10.610733 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:23:10.615552 (+ 4819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:10.615559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:10.615668 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:10.615745 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:10.617903 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:10.618862 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:10.620941 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:10.621317 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:10.621395 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:10.628789 (+ 7394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:10.628794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:10.628797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:10.628831 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:10.630578 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:10.631281 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:10.631285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:10.631287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:10.631369 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:10.631837 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:10.631841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:10.631843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:10.632089 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:10.632170 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:10.634006 (+ 1836us) server_negotiation.cc:299] Negotiation successful
0926 20:23:10.634162 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:23:10.636221 13581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:10.609466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44213 (local address 127.0.116.194:58099)
0926 20:23:10.610031 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:10.610090 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:10.611367 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:10.616331 (+ 4964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:10.616347 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:10.616797 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:10.617673 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:10.617702 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:10.619058 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:10.619064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:10.620788 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:10.620800 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:10.621613 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:10.621616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:10.621833 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:10.622787 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:10.622811 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:10.628509 (+ 5698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:10.630748 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:10.630756 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:10.630776 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:10.631162 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:10.631469 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:10.631472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:10.631474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:10.631726 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:10.632236 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:10.632243 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:10.632602 (+ 359us) client_negotiation.cc:769] Sending connection context
0926 20:23:10.634836 (+ 2234us) client_negotiation.cc:240] Negotiation successful
0926 20:23:10.635200 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":169,"threads_started":1}
I20260926 20:23:10.637816 13578 heartbeater.cc:347] Connected to a master server at 127.0.116.254:44213
I20260926 20:23:10.638067 13578 heartbeater.cc:464] Registering TS with master...
I20260926 20:23:10.638661 13578 heartbeater.cc:511] Master 127.0.116.254:44213 requested a full tablet report, sending...
W20260926 20:23:10.640318 13578 heartbeater.cc:644] Failed to heartbeat to 127.0.116.254:44213 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 20:23:10.640435 13578 heartbeater.cc:464] Registering TS with master...
I20260926 20:23:10.640594 13578 heartbeater.cc:511] Master 127.0.116.254:44213 requested a full tablet report, sending...
I20260926 20:23:10.641445 13578 heartbeater.cc:464] Registering TS with master...
I20260926 20:23:10.641599 13578 heartbeater.cc:511] Master 127.0.116.254:44213 requested a full tablet report, sending...
W20260926 20:23:10.642355 13578 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:23:10.756163 13579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:10.756546 13579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:10.756615 13579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:10.761683 13579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:10.761757 13579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:10.761826 13579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:23:10.768760 13579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.116.212:40019/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:45625
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=38947
--tserver_master_addrs=127.0.116.254:44213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:10.770303 13579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:10.771476 13579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:10.782454 13587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:10.782774 13589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:10.782454 13586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:10.784278 13579 server_base.cc:1034] running on GCE node
I20260926 20:23:10.784773 13579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:10.785529 13579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:10.786755 13579 hybrid_clock.cc:648] HybridClock initialized: now 1790454190786713 us; error 63 us; skew 500 ppm
Sep 26 20:23:10 dist-test-slave-qsm9 krb5kdc[12247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:10.791589 13579 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:23:10.793862 13579 webserver.cc:460] Webserver started at http://127.0.116.195:38947/ using document root <none> and password file <none>
I20260926 20:23:10.794693 13579 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:10.794852 13579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:23:10 dist-test-slave-qsm9 krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
I20260926 20:23:10.822890 13579 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:23:10.826051 13597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:10.827775 13579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:23:10.827903 13579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "3ab62b6703d84f7b82dcf54023c46c83"
format_stamp: "Formatted at 2026-09-26 20:23:09 on dist-test-slave-qsm9"
server_key: "968d8617d30ca5d969a9dfd71283542f"
server_key_iv: "157ec7564605026f472fae7595611654"
server_key_version: "kuduclusterkey@0"
I20260926 20:23:10.828634 13579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:10.850033 13579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:10.853679 13579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:10.853919 13579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:10.854764 13579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:23:10.855788 13579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:23:10.855863 13579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:10.855938 13579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:23:10.855993 13579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:10.869282 13579 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:45625
I20260926 20:23:10.869329 13717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:45625 every 8 connection(s)
I20260926 20:23:10.870779 13579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:23:10.878400 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 13579
Sep 26 20:23:10 dist-test-slave-qsm9 krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:23:10.887672 13446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:10.873076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:58575 (local address 127.0.116.254:44213)
0926 20:23:10.873263 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:10.873269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:10.874203 (+ 934us) server_negotiation.cc:407] Connection header received
0926 20:23:10.875356 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:10.875360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:10.875417 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:10.875568 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:10.876774 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:10.877697 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:10.879131 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:10.879450 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:10.879587 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:10.883133 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:10.883137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:10.883141 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:10.883169 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:10.884847 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:10.885574 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:10.885579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:10.885581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:10.885698 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:10.886158 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:10.886162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:10.886164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:10.886401 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:10.886503 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:10.887252 (+ 749us) server_negotiation.cc:299] Negotiation successful
0926 20:23:10.887434 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 20:23:10.889405 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:10.873506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44213 (local address 127.0.116.195:58575)
0926 20:23:10.874012 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:10.874066 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:10.875095 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:10.875779 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:10.875786 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:10.876106 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:10.876606 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:10.876619 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:10.877857 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:10.877860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:10.878969 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:10.878985 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:10.879790 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:10.879793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:10.879939 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:10.881339 (+ 1400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:10.881357 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:10.882942 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:10.885015 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:10.885022 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:10.885039 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:10.885422 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:10.885824 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:10.885827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:10.885829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:10.886032 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:10.886579 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:10.886587 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:10.886931 (+ 344us) client_negotiation.cc:769] Sending connection context
0926 20:23:10.887869 (+ 938us) client_negotiation.cc:240] Negotiation successful
0926 20:23:10.888256 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":143,"threads_started":1}
I20260926 20:23:10.890817 13718 heartbeater.cc:347] Connected to a master server at 127.0.116.254:44213
I20260926 20:23:10.891142 13718 heartbeater.cc:464] Registering TS with master...
I20260926 20:23:10.891836 13718 heartbeater.cc:511] Master 127.0.116.254:44213 requested a full tablet report, sending...
I20260926 20:23:10.893486 13244 ts_manager.cc:194] Registered new tserver with Master: 3ab62b6703d84f7b82dcf54023c46c83 (127.0.116.195:45625)
I20260926 20:23:10.894122 13721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:10.879184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35194 (local address 127.0.116.254:44213)
0926 20:23:10.880159 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:10.880168 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:10.880181 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:23:10.880216 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:10.880219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:10.880263 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:10.880365 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:10.882031 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:10.885921 (+ 3890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:10.887600 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:10.888281 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:10.888389 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:10.890364 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:10.890368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:10.890371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:10.890399 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:10.892077 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:10.892621 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:10.892625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:10.892627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:10.892727 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:10.893083 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:10.893087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:10.893089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:10.893292 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:10.893515 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:10.893776 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 20:23:10.893911 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":103,"threads_started":1}
I20260926 20:23:10.895231 13244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:58575
Sep 26 20:23:11 dist-test-slave-qsm9 krb5kdc[12247](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790454191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:11.375061 13442 heartbeater.cc:464] Registering TS with master...
I20260926 20:23:11.375404 13442 heartbeater.cc:511] Master 127.0.116.254:44213 requested a full tablet report, sending...
I20260926 20:23:11.376214 13244 ts_manager.cc:194] Registered new tserver with Master: 3981288b43eb4f7d82168b892110eeef (127.0.116.193:46333)
I20260926 20:23:11.377074 13244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:42115
Sep 26 20:23:11 dist-test-slave-qsm9 krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
I20260926 20:23:11.642575 13578 heartbeater.cc:464] Registering TS with master...
I20260926 20:23:11.642891 13578 heartbeater.cc:511] Master 127.0.116.254:44213 requested a full tablet report, sending...
I20260926 20:23:11.643905 13244 ts_manager.cc:194] Registered new tserver with Master: da1339f6b1ae4efd859146c265b90ffb (127.0.116.194:36859)
I20260926 20:23:11.644690 13244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:58099
I20260926 20:23:11.646281 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:23:11.665548 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:11.648218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57718 (local address 127.0.116.254:44213)
0926 20:23:11.652039 (+ 3821us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:11.652043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:11.652053 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:23:11.652088 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:11.652090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:11.652128 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:11.652231 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:11.653090 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:11.653930 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:11.655137 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:11.655540 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:11.655637 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:11.662041 (+ 6404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:11.662044 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:11.662047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:11.662081 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:11.663752 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:11.664240 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:11.664244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:11.664245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:11.664326 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:11.664615 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:11.664619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:11.664620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:11.664824 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:11.664969 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:11.665206 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 20:23:11.665347 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3754,"thread_start_us":92,"threads_started":1}
I20260926 20:23:11.671010 13244 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:23:11.674760 13244 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:23:11.698828 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:11.686926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:36859 (local address 127.0.0.1:41456)
0926 20:23:11.692674 (+ 5748us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:11.692686 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:11.692770 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:11.693403 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:11.693406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:11.693421 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:23:11.693499 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:11.693504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:11.694824 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:11.694827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:11.696914 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:11.696924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:11.698274 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:11.698278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:11.698396 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:11.698435 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 20:23:11.698545 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 20:23:11.698670 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5689,"thread_start_us":94,"threads_started":1}
I20260926 20:23:11.699452 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:11.687336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41456 (local address 127.0.116.194:36859)
0926 20:23:11.687620 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:11.687629 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:11.692870 (+ 5241us) server_negotiation.cc:407] Connection header received
0926 20:23:11.692952 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:11.692960 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:11.693132 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:11.693268 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:23:11.693618 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:11.694698 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:11.697048 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:11.697940 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:11.698039 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:11.698201 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:11.699188 (+ 987us) server_negotiation.cc:299] Negotiation successful
0926 20:23:11.699320 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":112,"threads_started":1}
I20260926 20:23:11.702040 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:11.685054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:46333 (local address 127.0.0.1:54976)
0926 20:23:11.691826 (+ 6772us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:11.691869 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:11.692029 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:11.692855 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:11.692857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:11.692872 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:23:11.693206 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:11.693211 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:11.697200 (+ 3989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:11.697203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:11.698279 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:11.698289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:11.700681 (+ 2392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:11.700685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:11.700774 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:11.700796 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:23:11.701819 (+ 1023us) client_negotiation.cc:240] Negotiation successful
0926 20:23:11.701903 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6573,"mutex_wait_us":215,"spinlock_wait_cycles":6400,"thread_start_us":97,"threads_started":1}
I20260926 20:23:11.702683 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:11.684971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:45625 (local address 127.0.0.1:47142)
0926 20:23:11.691792 (+ 6821us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:11.691861 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:11.692028 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:11.692823 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:11.692829 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:11.692852 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:23:11.693054 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:11.693064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:11.694530 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:11.694533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:11.695834 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:11.695845 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:11.702305 (+ 6460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:11.702309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:11.702399 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:11.702421 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:23:11.702492 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 20:23:11.702559 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6618,"mutex_wait_us":1,"spinlock_wait_cycles":9856,"thread_start_us":112,"threads_started":1}
I20260926 20:23:11.703629 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:11.686203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54976 (local address 127.0.116.193:46333)
0926 20:23:11.687712 (+ 1509us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:11.687717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:11.692176 (+ 4459us) server_negotiation.cc:407] Connection header received
0926 20:23:11.692265 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:11.692270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:11.692520 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:11.692667 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:23:11.695989 (+ 3322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:11.697055 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:11.699577 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:11.700386 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:11.700488 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:11.700616 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:11.703328 (+ 2712us) server_negotiation.cc:299] Negotiation successful
0926 20:23:11.703458 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":996,"thread_start_us":115,"threads_started":1}
I20260926 20:23:11.703455 13509 tablet_service.cc:1467] Processing CreateTablet for tablet 6c0c1dc64be8467ab025c8cca6560c10 (DEFAULT_TABLE table=test-table [id=9e7e046485db43b5a9079e51dd96dfff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:23:11.704897 13509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c0c1dc64be8467ab025c8cca6560c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:23:11.705191 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:11.685816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47142 (local address 127.0.116.195:45625)
0926 20:23:11.687119 (+ 1303us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:11.687127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:11.692196 (+ 5069us) server_negotiation.cc:407] Connection header received
0926 20:23:11.692287 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:11.692293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:11.692504 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:11.692646 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:23:11.693325 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:11.694401 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:11.698430 (+ 4029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:11.699254 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:11.699356 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:11.699515 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:11.704892 (+ 5377us) server_negotiation.cc:299] Negotiation successful
0926 20:23:11.705057 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1185,"thread_start_us":107,"threads_started":1}
I20260926 20:23:11.707058 13375 tablet_service.cc:1467] Processing CreateTablet for tablet 6c0c1dc64be8467ab025c8cca6560c10 (DEFAULT_TABLE table=test-table [id=9e7e046485db43b5a9079e51dd96dfff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:23:11.708242 13375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c0c1dc64be8467ab025c8cca6560c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:23:11.708033 13640 tablet_service.cc:1467] Processing CreateTablet for tablet 6c0c1dc64be8467ab025c8cca6560c10 (DEFAULT_TABLE table=test-table [id=9e7e046485db43b5a9079e51dd96dfff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:23:11.708818 13640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c0c1dc64be8467ab025c8cca6560c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:23:11.715188 13744 tablet_bootstrap.cc:492] T 6c0c1dc64be8467ab025c8cca6560c10 P da1339f6b1ae4efd859146c265b90ffb: Bootstrap starting.
I20260926 20:23:11.717626 13744 tablet_bootstrap.cc:654] T 6c0c1dc64be8467ab025c8cca6560c10 P da1339f6b1ae4efd859146c265b90ffb: Neither blocks nor log segments found. Creating new log.
I20260926 20:23:11.718425 13744 log.cc:824] T 6c0c1dc64be8467ab025c8cca6560c10 P da1339f6b1ae4efd859146c265b90ffb: Log is configured to *not* fsync() on all Append() calls
I20260926 20:23:11.719627 13746 tablet_bootstrap.cc:492] T 6c0c1dc64be8467ab025c8cca6560c10 P 3981288b43eb4f7d82168b892110eeef: Bootstrap starting.
I20260926 20:23:11.719923 13745 tablet_bootstrap.cc:492] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83: Bootstrap starting.
I20260926 20:23:11.720947 13744 tablet_bootstrap.cc:492] T 6c0c1dc64be8467ab025c8cca6560c10 P da1339f6b1ae4efd859146c265b90ffb: No bootstrap required, opened a new log
I20260926 20:23:11.721215 13744 ts_tablet_manager.cc:1397] T 6c0c1dc64be8467ab025c8cca6560c10 P da1339f6b1ae4efd859146c265b90ffb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260926 20:23:11.722121 13746 tablet_bootstrap.cc:654] T 6c0c1dc64be8467ab025c8cca6560c10 P 3981288b43eb4f7d82168b892110eeef: Neither blocks nor log segments found. Creating new log.
I20260926 20:23:11.722177 13745 tablet_bootstrap.cc:654] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83: Neither blocks nor log segments found. Creating new log.
I20260926 20:23:11.723078 13745 log.cc:824] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83: Log is configured to *not* fsync() on all Append() calls
I20260926 20:23:11.723486 13746 log.cc:824] T 6c0c1dc64be8467ab025c8cca6560c10 P 3981288b43eb4f7d82168b892110eeef: Log is configured to *not* fsync() on all Append() calls
I20260926 20:23:11.725821 13745 tablet_bootstrap.cc:492] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83: No bootstrap required, opened a new log
I20260926 20:23:11.725821 13746 tablet_bootstrap.cc:492] T 6c0c1dc64be8467ab025c8cca6560c10 P 3981288b43eb4f7d82168b892110eeef: No bootstrap required, opened a new log
I20260926 20:23:11.726033 13746 ts_tablet_manager.cc:1397] T 6c0c1dc64be8467ab025c8cca6560c10 P 3981288b43eb4f7d82168b892110eeef: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 20:23:11.725839 13744 raft_consensus.cc:348] T 6c0c1dc64be8467ab025c8cca6560c10 P da1339f6b1ae4efd859146c265b90ffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab62b6703d84f7b82dcf54023c46c83" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 45625 } } peers { permanent_uuid: "3981288b43eb4f7d82168b892110eeef" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 46333 } } peers { permanent_uuid: "da1339f6b1ae4efd859146c265b90ffb" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 36859 } }
I20260926 20:23:11.726049 13745 ts_tablet_manager.cc:1397] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20260926 20:23:11.726114 13744 raft_consensus.cc:374] T 6c0c1dc64be8467ab025c8cca6560c10 P da1339f6b1ae4efd859146c265b90ffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:23:11.726181 13744 raft_consensus.cc:729] T 6c0c1dc64be8467ab025c8cca6560c10 P da1339f6b1ae4efd859146c265b90ffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da1339f6b1ae4efd859146c265b90ffb, State: Initialized, Role: FOLLOWER
I20260926 20:23:11.726764 13744 consensus_queue.cc:260] T 6c0c1dc64be8467ab025c8cca6560c10 P da1339f6b1ae4efd859146c265b90ffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab62b6703d84f7b82dcf54023c46c83" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 45625 } } peers { permanent_uuid: "3981288b43eb4f7d82168b892110eeef" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 46333 } } peers { permanent_uuid: "da1339f6b1ae4efd859146c265b90ffb" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 36859 } }
I20260926 20:23:11.727906 13744 ts_tablet_manager.cc:1428] T 6c0c1dc64be8467ab025c8cca6560c10 P da1339f6b1ae4efd859146c265b90ffb: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260926 20:23:11.728287 13578 heartbeater.cc:503] Master 127.0.116.254:44213 was elected leader, sending a full tablet report...
I20260926 20:23:11.730310 13746 raft_consensus.cc:348] T 6c0c1dc64be8467ab025c8cca6560c10 P 3981288b43eb4f7d82168b892110eeef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab62b6703d84f7b82dcf54023c46c83" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 45625 } } peers { permanent_uuid: "3981288b43eb4f7d82168b892110eeef" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 46333 } } peers { permanent_uuid: "da1339f6b1ae4efd859146c265b90ffb" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 36859 } }
I20260926 20:23:11.730593 13746 raft_consensus.cc:374] T 6c0c1dc64be8467ab025c8cca6560c10 P 3981288b43eb4f7d82168b892110eeef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:23:11.730669 13746 raft_consensus.cc:729] T 6c0c1dc64be8467ab025c8cca6560c10 P 3981288b43eb4f7d82168b892110eeef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3981288b43eb4f7d82168b892110eeef, State: Initialized, Role: FOLLOWER
I20260926 20:23:11.730602 13745 raft_consensus.cc:348] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab62b6703d84f7b82dcf54023c46c83" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 45625 } } peers { permanent_uuid: "3981288b43eb4f7d82168b892110eeef" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 46333 } } peers { permanent_uuid: "da1339f6b1ae4efd859146c265b90ffb" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 36859 } }
I20260926 20:23:11.730839 13745 raft_consensus.cc:374] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:23:11.730881 13745 raft_consensus.cc:729] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab62b6703d84f7b82dcf54023c46c83, State: Initialized, Role: FOLLOWER
I20260926 20:23:11.731261 13746 consensus_queue.cc:260] T 6c0c1dc64be8467ab025c8cca6560c10 P 3981288b43eb4f7d82168b892110eeef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab62b6703d84f7b82dcf54023c46c83" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 45625 } } peers { permanent_uuid: "3981288b43eb4f7d82168b892110eeef" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 46333 } } peers { permanent_uuid: "da1339f6b1ae4efd859146c265b90ffb" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 36859 } }
I20260926 20:23:11.731323 13745 consensus_queue.cc:260] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab62b6703d84f7b82dcf54023c46c83" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 45625 } } peers { permanent_uuid: "3981288b43eb4f7d82168b892110eeef" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 46333 } } peers { permanent_uuid: "da1339f6b1ae4efd859146c265b90ffb" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 36859 } }
I20260926 20:23:11.732364 13745 ts_tablet_manager.cc:1428] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260926 20:23:11.732365 13746 ts_tablet_manager.cc:1428] T 6c0c1dc64be8467ab025c8cca6560c10 P 3981288b43eb4f7d82168b892110eeef: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260926 20:23:11.732750 13718 heartbeater.cc:503] Master 127.0.116.254:44213 was elected leader, sending a full tablet report...
I20260926 20:23:11.732749 13442 heartbeater.cc:503] Master 127.0.116.254:44213 was elected leader, sending a full tablet report...
W20260926 20:23:11.833850 13443 tablet.cc:2369] T 6c0c1dc64be8467ab025c8cca6560c10 P 3981288b43eb4f7d82168b892110eeef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:23:11.862820 13580 tablet.cc:2369] T 6c0c1dc64be8467ab025c8cca6560c10 P da1339f6b1ae4efd859146c265b90ffb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:23:11.872570 13719 tablet.cc:2369] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 20:23:11 dist-test-slave-qsm9 krb5kdc[12247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for HTTP/127.0.116.212@KRBTEST.COM
I20260926 20:23:12.017534 13752 raft_consensus.cc:482] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:23:12.017917 13752 raft_consensus.cc:504] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab62b6703d84f7b82dcf54023c46c83" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 45625 } } peers { permanent_uuid: "3981288b43eb4f7d82168b892110eeef" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 46333 } } peers { permanent_uuid: "da1339f6b1ae4efd859146c265b90ffb" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 36859 } }
I20260926 20:23:12.019635 13752 leader_election.cc:288] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3981288b43eb4f7d82168b892110eeef (127.0.116.193:46333), da1339f6b1ae4efd859146c265b90ffb (127.0.116.194:36859)
I20260926 20:23:12.029875 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:12.019843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:46333 (local address 127.0.116.195:47965)
0926 20:23:12.021253 (+ 1410us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:12.021270 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:12.021459 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:12.021895 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:12.021898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:12.021919 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:23:12.022032 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:12.022038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:12.023188 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:12.023192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:12.024629 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:12.024642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:12.029375 (+ 4733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:12.029379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:12.029482 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:12.029508 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 20:23:12.029587 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 20:23:12.029673 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1328,"thread_start_us":137,"threads_started":1}
I20260926 20:23:12.029899 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:12.020857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:47965 (local address 127.0.116.193:46333)
0926 20:23:12.021004 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:12.021009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:12.021356 (+ 347us) server_negotiation.cc:407] Connection header received
0926 20:23:12.021597 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:12.021601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:12.021653 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:12.021734 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:23:12.022191 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:12.023012 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:12.028404 (+ 5392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:12.029086 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:12.029212 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:12.029270 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:12.029665 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 20:23:12.029765 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:23:12.031194 13394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c0c1dc64be8467ab025c8cca6560c10" candidate_uuid: "3ab62b6703d84f7b82dcf54023c46c83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3981288b43eb4f7d82168b892110eeef" is_pre_election: true
I20260926 20:23:12.031584 13394 raft_consensus.cc:2454] T 6c0c1dc64be8467ab025c8cca6560c10 P 3981288b43eb4f7d82168b892110eeef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ab62b6703d84f7b82dcf54023c46c83 in term 0.
I20260926 20:23:12.032179 13600 leader_election.cc:302] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3981288b43eb4f7d82168b892110eeef, 3ab62b6703d84f7b82dcf54023c46c83; no voters:
I20260926 20:23:12.032518 13752 raft_consensus.cc:2781] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:23:12.032595 13752 raft_consensus.cc:482] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:23:12.032637 13752 raft_consensus.cc:3037] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:23:12.034181 13752 raft_consensus.cc:504] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab62b6703d84f7b82dcf54023c46c83" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 45625 } } peers { permanent_uuid: "3981288b43eb4f7d82168b892110eeef" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 46333 } } peers { permanent_uuid: "da1339f6b1ae4efd859146c265b90ffb" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 36859 } }
I20260926 20:23:12.034595 13752 leader_election.cc:288] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83 [CANDIDATE]: Term 1 election: Requested vote from peers 3981288b43eb4f7d82168b892110eeef (127.0.116.193:46333), da1339f6b1ae4efd859146c265b90ffb (127.0.116.194:36859)
I20260926 20:23:12.035410 13394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c0c1dc64be8467ab025c8cca6560c10" candidate_uuid: "3ab62b6703d84f7b82dcf54023c46c83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3981288b43eb4f7d82168b892110eeef"
I20260926 20:23:12.035574 13394 raft_consensus.cc:3037] T 6c0c1dc64be8467ab025c8cca6560c10 P 3981288b43eb4f7d82168b892110eeef [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:23:12.037017 13394 raft_consensus.cc:2454] T 6c0c1dc64be8467ab025c8cca6560c10 P 3981288b43eb4f7d82168b892110eeef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ab62b6703d84f7b82dcf54023c46c83 in term 1.
I20260926 20:23:12.037508 13600 leader_election.cc:302] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3981288b43eb4f7d82168b892110eeef, 3ab62b6703d84f7b82dcf54023c46c83; no voters:
I20260926 20:23:12.037786 13752 raft_consensus.cc:2781] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:23:12.038023 13752 raft_consensus.cc:686] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83 [term 1 LEADER]: Becoming Leader. State: Replica: 3ab62b6703d84f7b82dcf54023c46c83, State: Running, Role: LEADER
I20260926 20:23:12.038424 13752 consensus_queue.cc:237] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab62b6703d84f7b82dcf54023c46c83" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 45625 } } peers { permanent_uuid: "3981288b43eb4f7d82168b892110eeef" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 46333 } } peers { permanent_uuid: "da1339f6b1ae4efd859146c265b90ffb" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 36859 } }
I20260926 20:23:12.042291 13239 catalog_manager.cc:5495] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ab62b6703d84f7b82dcf54023c46c83 (127.0.116.195). New cstate: current_term: 1 leader_uuid: "3ab62b6703d84f7b82dcf54023c46c83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab62b6703d84f7b82dcf54023c46c83" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 45625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3981288b43eb4f7d82168b892110eeef" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 46333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da1339f6b1ae4efd859146c265b90ffb" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 36859 } health_report { overall_health: UNKNOWN } } }
I20260926 20:23:12.046330 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:12.021035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:43653 (local address 127.0.116.194:36859)
0926 20:23:12.021154 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:12.021158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:12.025045 (+ 3887us) server_negotiation.cc:407] Connection header received
0926 20:23:12.025270 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:12.025274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:12.025328 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:12.025381 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:23:12.025805 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:12.026637 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:12.043767 (+ 17130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:12.044542 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:12.044678 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:12.044740 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:12.046030 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0926 20:23:12.046164 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:23:12.047113 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:12.020300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:36859 (local address 127.0.116.195:43653)
0926 20:23:12.024946 (+ 4646us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:12.024963 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:12.025139 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:12.025521 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:12.025524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:12.025542 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:23:12.025663 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:12.025669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:12.026788 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:12.026792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:12.028224 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:12.028236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:12.045704 (+ 17468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:12.045711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:12.045841 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:12.045874 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 20:23:12.046835 (+ 961us) client_negotiation.cc:240] Negotiation successful
0926 20:23:12.046943 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4560,"thread_start_us":85,"threads_started":1}
I20260926 20:23:12.048362 13516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c0c1dc64be8467ab025c8cca6560c10" candidate_uuid: "3ab62b6703d84f7b82dcf54023c46c83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da1339f6b1ae4efd859146c265b90ffb"
I20260926 20:23:12.048669 13516 raft_consensus.cc:3037] T 6c0c1dc64be8467ab025c8cca6560c10 P da1339f6b1ae4efd859146c265b90ffb [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:23:12.050289 13516 raft_consensus.cc:2454] T 6c0c1dc64be8467ab025c8cca6560c10 P da1339f6b1ae4efd859146c265b90ffb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ab62b6703d84f7b82dcf54023c46c83 in term 1.
I20260926 20:23:12.048382 13513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c0c1dc64be8467ab025c8cca6560c10" candidate_uuid: "3ab62b6703d84f7b82dcf54023c46c83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da1339f6b1ae4efd859146c265b90ffb" is_pre_election: true
I20260926 20:23:12.050725 13513 raft_consensus.cc:2362] T 6c0c1dc64be8467ab025c8cca6560c10 P da1339f6b1ae4efd859146c265b90ffb [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3ab62b6703d84f7b82dcf54023c46c83 in term 1. Re-sending same reply.
I20260926 20:23:12.060834 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:12.055921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47144 (local address 127.0.116.195:45625)
0926 20:23:12.056065 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:12.056070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:12.056084 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:23:12.056164 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:12.056167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:12.056234 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:12.056360 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:23:12.056759 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:12.057561 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:12.058824 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:12.059175 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:12.059298 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:12.059862 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:23:12.060335 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:23:12.060499 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:12.060578 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 20:23:12.060650 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:23:12.072989 13513 raft_consensus.cc:1261] T 6c0c1dc64be8467ab025c8cca6560c10 P da1339f6b1ae4efd859146c265b90ffb [term 1 FOLLOWER]: Refusing update from remote peer 3ab62b6703d84f7b82dcf54023c46c83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:23:12.074199 13757 consensus_queue.cc:1035] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da1339f6b1ae4efd859146c265b90ffb" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 36859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:23:12.074445 13394 raft_consensus.cc:1261] T 6c0c1dc64be8467ab025c8cca6560c10 P 3981288b43eb4f7d82168b892110eeef [term 1 FOLLOWER]: Refusing update from remote peer 3ab62b6703d84f7b82dcf54023c46c83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:23:12.075548 13757 consensus_queue.cc:1035] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3981288b43eb4f7d82168b892110eeef" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 46333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:23:12.085433 13760 mvcc.cc:205] Tried to move back new op lower bound from 7333700370718375936 to 7333700370593202176. Current Snapshot: MvccSnapshot[applied={T|T < 7333700370718375936}]
I20260926 20:23:12.087119 13761 mvcc.cc:205] Tried to move back new op lower bound from 7333700370718375936 to 7333700370593202176. Current Snapshot: MvccSnapshot[applied={T|T < 7333700370718375936}]
I20260926 20:23:12.101366 13239 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:23:12.101568 13239 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:23:12.104636 13239 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39d341d5c84248d282e716ce2544f884: Sending DeleteTablet for 3 replicas of tablet 6c0c1dc64be8467ab025c8cca6560c10
I20260926 20:23:12.105580 13640 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c0c1dc64be8467ab025c8cca6560c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:23:12 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:47142
I20260926 20:23:12.106812 13375 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c0c1dc64be8467ab025c8cca6560c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:23:12 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:54976
I20260926 20:23:12.107405 13509 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c0c1dc64be8467ab025c8cca6560c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:23:12 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:41456
I20260926 20:23:12.110163 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 13282
I20260926 20:23:12.115798 13769 tablet_replica.cc:330] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83: stopping tablet replica
I20260926 20:23:12.116225 13769 raft_consensus.cc:2229] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:23:12.116581 13771 tablet_replica.cc:330] T 6c0c1dc64be8467ab025c8cca6560c10 P da1339f6b1ae4efd859146c265b90ffb: stopping tablet replica
I20260926 20:23:12.118427 13771 raft_consensus.cc:2229] T 6c0c1dc64be8467ab025c8cca6560c10 P da1339f6b1ae4efd859146c265b90ffb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:23:12.118868 13771 raft_consensus.cc:2258] T 6c0c1dc64be8467ab025c8cca6560c10 P da1339f6b1ae4efd859146c265b90ffb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:23:12.119561 13769 raft_consensus.cc:2258] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:23:12.121750 13769 ts_tablet_manager.cc:1905] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:23:12.121814 13771 ts_tablet_manager.cc:1905] T 6c0c1dc64be8467ab025c8cca6560c10 P da1339f6b1ae4efd859146c265b90ffb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 20:23:12.122913 13230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.116.193:46333 (error 108)
I20260926 20:23:12.124106 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 13445
I20260926 20:23:12.128970 13769 ts_tablet_manager.cc:1918] T 6c0c1dc64be8467ab025c8cca6560c10 P 3ab62b6703d84f7b82dcf54023c46c83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:23:12.130815 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 13579
I20260926 20:23:12.133956 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:12.133730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:36859 (local address 127.0.0.1:41460)
0926 20:23:12.133867 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:12.133906 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.194:36859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 20:23:12.133956 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:12.133464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:46333 (local address 127.0.0.1:54988)
0926 20:23:12.133768 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:12.133852 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.193:46333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206}
W20260926 20:23:12.134517 13230 catalog_manager.cc:4584] TS da1339f6b1ae4efd859146c265b90ffb (127.0.116.194:36859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c0c1dc64be8467ab025c8cca6560c10: Network error: Client connection negotiation failed: client connection to 127.0.116.194:36859: connect: Connection refused (error 111)
I20260926 20:23:12.138824 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 13211
I20260926 20:23:12.147784 467 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 20:23:12.483385 467 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 20:23:12.483570 467 mini_ranger.cc:66] Stopping Ranger...
I20260926 20:23:12.829345 467 mini_ranger.cc:68] Stopped Ranger
2026-09-26 20:23:12.829 UTC [12277] LOG: received smart shutdown request
2026-09-26 20:23:12.833 UTC [12277] LOG: background worker "logical replication launcher" (PID 12286) exited with exit code 1
2026-09-26 20:23:12.834 UTC [12281] LOG: shutting down
2026-09-26 20:23:13.038 UTC [12277] LOG: database system is shut down
2026-09-26T20:23:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49841 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:23:13 dist-test-slave-qsm9 krb5kdc[13781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:23:13 dist-test-slave-qsm9 krb5kdc[13781](info): set up 2 sockets
Sep 26 20:23:13 dist-test-slave-qsm9 krb5kdc[13781](info): commencing operation
krb5kdc: starting...
W20260926 20:23:14.216161 467 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:23:14 dist-test-slave-qsm9 krb5kdc[13781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:23:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:23:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:23:14.343035 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:43525
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:44333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:43525,127.0.116.253:44509,127.0.116.252:42171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:23:14.458593 13794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:14.458948 13794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:14.459013 13794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:14.463202 13794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:23:14.463266 13794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:14.463287 13794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:14.463304 13794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:23:14.463320 13794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:23:14.468731 13794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:44333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:43525,127.0.116.253:44509,127.0.116.252:42171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:43525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:14.469906 13794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:14.470933 13794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:14.480412 13800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:14.480602 13799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:14.480782 13794 server_base.cc:1034] running on GCE node
W20260926 20:23:14.480458 13802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:14.481451 13794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:14.483089 13794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:14.484246 13794 hybrid_clock.cc:648] HybridClock initialized: now 1790454194484223 us; error 40 us; skew 500 ppm
Sep 26 20:23:14 dist-test-slave-qsm9 krb5kdc[13781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:14.488085 13794 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:23:14.489009 13794 webserver.cc:460] Webserver started at http://127.0.116.254:44179/ using document root <none> and password file <none>
I20260926 20:23:14.489629 13794 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:14.489732 13794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:14.489952 13794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:23:14.492133 13794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "b66a9817d6e641cca8fd1176b492fa42"
format_stamp: "Formatted at 2026-09-26 20:23:14 on dist-test-slave-qsm9"
server_key: "20e2133730661af47678759179e76cb8"
server_key_iv: "6acd9c9055856343f16e62b2f2d6fb5c"
server_key_version: "encryptionkey@0"
I20260926 20:23:14.492727 13794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "b66a9817d6e641cca8fd1176b492fa42"
format_stamp: "Formatted at 2026-09-26 20:23:14 on dist-test-slave-qsm9"
server_key: "20e2133730661af47678759179e76cb8"
server_key_iv: "6acd9c9055856343f16e62b2f2d6fb5c"
server_key_version: "encryptionkey@0"
I20260926 20:23:14.496572 13794 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:23:14.499207 13809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:14.500736 13794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:23:14.500892 13794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "b66a9817d6e641cca8fd1176b492fa42"
format_stamp: "Formatted at 2026-09-26 20:23:14 on dist-test-slave-qsm9"
server_key: "20e2133730661af47678759179e76cb8"
server_key_iv: "6acd9c9055856343f16e62b2f2d6fb5c"
server_key_version: "encryptionkey@0"
I20260926 20:23:14.501019 13794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:14.525365 13794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:14.528648 13794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:14.528895 13794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:14.536203 13794 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:43525
I20260926 20:23:14.536281 13861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:43525 every 8 connection(s)
I20260926 20:23:14.537302 13794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:23:14.540257 13862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:23:14.540483 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 13794
I20260926 20:23:14.540624 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:23:14.540943 467 external_mini_cluster.cc:1428] Setting key 0ac8391d1a4c30de5c525fbb53cd4692
I20260926 20:23:14.544799 13862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43525 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.116.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.253@KRBTEST.COM" created.
Sep 26 20:23:14 dist-test-slave-qsm9 krb5kdc[13781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:23:14.561084 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.547201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41456 (local address 127.0.116.254:43525)
0926 20:23:14.547678 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:14.547698 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:14.547974 (+ 276us) server_negotiation.cc:407] Connection header received
0926 20:23:14.548979 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:14.548983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:14.549163 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:14.549353 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:14.550191 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:14.551172 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:14.552560 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:14.552946 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:14.553129 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:14.555959 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:14.555967 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:14.555983 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:14.556008 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:14.558032 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:14.558535 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:14.558543 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:14.558545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:14.558696 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:14.559014 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:14.559019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:14.559021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:14.559619 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:14.559775 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:14.560128 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 20:23:14.560334 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":308,"thread_start_us":151,"threads_started":1}
I20260926 20:23:14.561088 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.547201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:43525 (local address 127.0.0.1:41456)
0926 20:23:14.547825 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.547861 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:14.548670 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:14.549348 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:14.549353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:14.549769 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:14.550048 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:14.550061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:14.551290 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:14.551294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:14.552423 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:14.552437 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:14.553123 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:14.553129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:14.553371 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:14.553988 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:14.554004 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:14.555793 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:14.558158 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:14.558166 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:14.558174 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:14.558402 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:14.558793 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:14.558795 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:14.558796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:14.558902 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:14.559735 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:14.559740 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:14.559859 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 20:23:14.559993 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 20:23:14.560259 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":77,"thread_start_us":144,"threads_started":1}
I20260926 20:23:14.563452 13862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:23:14.564316 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.564050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:44509 (local address 127.0.0.1:55406)
0926 20:23:14.564165 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.564259 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.253:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 20:23:14.564554 13813 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.116.253:44509: connect: Connection refused (error 111)
I20260926 20:23:14.565366 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.565155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:44509 (local address 127.0.0.1:55420)
0926 20:23:14.565266 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.565316 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.253:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 20:23:14.565522 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.253:44509: Network error: Client connection negotiation failed: client connection to 127.0.116.253:44509: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.253.keytab.
Entry for principal kudu/127.0.116.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.253.keytab.
I20260926 20:23:14.571861 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.253:44509
--webserver_interface=127.0.116.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.253.keytab
--principal=kudu/127.0.116.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:44333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:43525,127.0.116.253:44509,127.0.116.252:42171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:23:14.614786 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } attempt: 1
I20260926 20:23:14.615806 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.615410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:44509 (local address 127.0.0.1:55434)
0926 20:23:14.615627 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.615710 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.253:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 20:23:14.616608 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.616376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:44509 (local address 127.0.0.1:55450)
0926 20:23:14.616485 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.616540 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.253:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 20:23:14.616758 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.253:44509: Network error: Client connection negotiation failed: client connection to 127.0.116.253:44509: connect: Connection refused (error 111)
I20260926 20:23:14.684969 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } attempt: 2
I20260926 20:23:14.685899 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.685529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:44509 (local address 127.0.0.1:55460)
0926 20:23:14.685716 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.685801 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.253:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 20:23:14.686682 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.686445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:44509 (local address 127.0.0.1:55462)
0926 20:23:14.686555 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.686612 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.253:44509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 20:23:14.686829 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.253:44509: Network error: Client connection negotiation failed: client connection to 127.0.116.253:44509: connect: Connection refused (error 111)
W20260926 20:23:14.687376 13869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:14.687741 13869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:14.687830 13869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:14.692560 13869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:23:14.692716 13869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:14.692759 13869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:14.692799 13869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:23:14.692839 13869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:23:14.698503 13869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:44333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:43525,127.0.116.253:44509,127.0.116.252:42171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.253.keytab
--principal=kudu/127.0.116.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.253:44509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.116.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:14.699733 13869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:14.700765 13869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:14.709597 13874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:14.709590 13877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:14.709590 13875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:14.709950 13869 server_base.cc:1034] running on GCE node
I20260926 20:23:14.710530 13869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:14.711789 13869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:14.712972 13869 hybrid_clock.cc:648] HybridClock initialized: now 1790454194712957 us; error 35 us; skew 500 ppm
Sep 26 20:23:14 dist-test-slave-qsm9 krb5kdc[13781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:14.715969 13869 init.cc:377] Logged in from keytab as kudu/127.0.116.253@KRBTEST.COM (short username kudu)
I20260926 20:23:14.716852 13869 webserver.cc:460] Webserver started at http://127.0.116.253:36931/ using document root <none> and password file <none>
I20260926 20:23:14.717374 13869 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:14.717432 13869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:14.717598 13869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:23:14.719758 13869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-1/data/instance:
uuid: "c084f09d19f44d33acc3b7c03747f9e2"
format_stamp: "Formatted at 2026-09-26 20:23:14 on dist-test-slave-qsm9"
server_key: "ead9b399db56c4cfc2d577447bbf0ff6"
server_key_iv: "3236565301ad8d4410fe488ea3f2163e"
server_key_version: "encryptionkey@0"
I20260926 20:23:14.720286 13869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-1/wal/instance:
uuid: "c084f09d19f44d33acc3b7c03747f9e2"
format_stamp: "Formatted at 2026-09-26 20:23:14 on dist-test-slave-qsm9"
server_key: "ead9b399db56c4cfc2d577447bbf0ff6"
server_key_iv: "3236565301ad8d4410fe488ea3f2163e"
server_key_version: "encryptionkey@0"
I20260926 20:23:14.724064 13869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:23:14.726579 13884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:14.728011 13869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:23:14.728149 13869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-1/wal
uuid: "c084f09d19f44d33acc3b7c03747f9e2"
format_stamp: "Formatted at 2026-09-26 20:23:14 on dist-test-slave-qsm9"
server_key: "ead9b399db56c4cfc2d577447bbf0ff6"
server_key_iv: "3236565301ad8d4410fe488ea3f2163e"
server_key_version: "encryptionkey@0"
I20260926 20:23:14.728266 13869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:14.743817 13869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:14.746798 13869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:14.747018 13869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:14.754262 13869 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.253:44509
I20260926 20:23:14.754279 13936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.253:44509 every 8 connection(s)
I20260926 20:23:14.755292 13869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-1/data/info.pb
I20260926 20:23:14.758241 13937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:23:14.758447 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 13869
I20260926 20:23:14.758570 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-1/wal/instance
I20260926 20:23:14.758876 467 external_mini_cluster.cc:1428] Setting key c0f399b3f17ceee5e8ff5d6e519525dc
I20260926 20:23:14.762862 13937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43525 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.116.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.252@KRBTEST.COM" created.
Sep 26 20:23:14 dist-test-slave-qsm9 krb5kdc[13781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.253@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:23:14.778080 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.765095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41468 (local address 127.0.116.254:43525)
0926 20:23:14.765245 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:14.765249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:14.766345 (+ 1096us) server_negotiation.cc:407] Connection header received
0926 20:23:14.767398 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:14.767405 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:14.767485 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:14.767618 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:14.768842 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:14.769366 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:14.770763 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:14.771013 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:14.771131 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:14.773942 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:14.773946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:14.773949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:14.773974 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:14.775717 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:14.776334 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:14.776339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:14.776342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:14.776430 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:14.776792 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:14.776796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:14.776797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:14.776992 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:14.777103 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:14.777756 (+ 653us) server_negotiation.cc:299] Negotiation successful
0926 20:23:14.777902 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:23:14.778018 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } attempt: 3
I20260926 20:23:14.779310 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.765614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:43525 (local address 127.0.0.1:41468)
0926 20:23:14.766175 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.766232 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:14.767087 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:14.767773 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:14.767781 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:14.768127 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:14.768688 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:14.768706 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:14.769483 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:14.769486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:14.770574 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:14.770585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:14.771187 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:14.771192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:14.771379 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:14.772059 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:14.772086 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:14.773756 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:14.775892 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:14.775899 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:14.775917 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:14.776199 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:14.776525 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:14.776528 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:14.776530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:14.776680 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:14.777119 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:14.777129 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:14.777490 (+ 361us) client_negotiation.cc:769] Sending connection context
0926 20:23:14.777698 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 20:23:14.778101 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":176,"threads_started":1}
I20260926 20:23:14.780925 13937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:23:14 dist-test-slave-qsm9 krb5kdc[13781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for kudu/127.0.116.253@KRBTEST.COM
Sep 26 20:23:14 dist-test-slave-qsm9 krb5kdc[13781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.253@KRBTEST.COM for kudu/127.0.116.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.252.keytab.
Entry for principal kudu/127.0.116.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.252.keytab.
I20260926 20:23:14.787204 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.252:42171
--webserver_interface=127.0.116.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.252.keytab
--principal=kudu/127.0.116.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:44333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:43525,127.0.116.253:44509,127.0.116.252:42171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:23:14.790934 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.778611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:44509 (local address 127.0.0.1:55470)
0926 20:23:14.778778 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.778796 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:14.778898 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:14.779614 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:14.779618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:14.779781 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:14.780476 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:14.780489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:14.781800 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:14.781803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:14.782644 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:14.782654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:14.783410 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:14.783414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:14.783565 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:14.784172 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:14.784189 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:14.785695 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:14.789259 (+ 3564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:14.789262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:14.789264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:14.789476 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:14.789799 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:14.789802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:14.789804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:14.789890 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:14.790366 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:14.790369 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:14.790479 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 20:23:14.790586 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 20:23:14.790734 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 20:23:14.791045 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.781432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:44509 (local address 127.0.0.1:55474)
0926 20:23:14.781545 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.781558 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:14.781627 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:14.782201 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:14.782203 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:14.782349 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:14.782435 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:14.782440 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:14.783261 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:14.783265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:14.783896 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:14.783907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:14.784483 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:14.784485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:14.784558 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:14.785044 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:14.785056 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:14.786410 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:14.789239 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:14.789244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:14.789247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:14.789532 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:14.789840 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:14.789842 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:14.789844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:14.789939 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:14.790430 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:14.790436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:14.790529 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 20:23:14.790626 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 20:23:14.790779 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 20:23:14.791052 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.778745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.0.116.253:44509)
0926 20:23:14.779118 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:14.779130 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:14.779159 (+ 29us) server_negotiation.cc:407] Connection header received
0926 20:23:14.779205 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:14.779210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:14.779423 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:14.780430 (+ 1007us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:14.780582 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:14.781671 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:14.782758 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:14.783287 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:14.783399 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:14.785839 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:14.785853 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:14.785870 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:14.785897 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:14.789173 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:14.789591 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:14.789598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:14.789600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:14.789700 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:14.789979 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:14.789984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:14.789986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:14.790246 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:14.790389 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:14.790683 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 20:23:14.790840 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":274,"spinlock_wait_cycles":3968,"thread_start_us":162,"threads_started":1}
I20260926 20:23:14.791685 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.781508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55474 (local address 127.0.116.253:44509)
0926 20:23:14.781983 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:14.781992 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:14.782005 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:23:14.782039 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:14.782042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:14.782093 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:14.782183 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:14.782544 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:14.783158 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:14.784028 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:14.784369 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:14.784479 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:14.786587 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:14.786592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:14.786595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:14.786627 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:14.789104 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:14.789653 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:14.789658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:14.789661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:14.789745 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:14.790036 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:14.790041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:14.790046 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:14.790276 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:14.790393 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:14.791270 (+ 877us) server_negotiation.cc:299] Negotiation successful
0926 20:23:14.791419 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":409,"thread_start_us":122,"threads_started":1}
I20260926 20:23:14.793304 13862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:23:14.793653 13937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:23:14.794131 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.793837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42171 (local address 127.0.0.1:51518)
0926 20:23:14.793990 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.794051 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 20:23:14.794396 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.794062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42171 (local address 127.0.0.1:51528)
0926 20:23:14.794205 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.794297 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 20:23:14.794620 13885 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.116.252:42171: connect: Connection refused (error 111)
I20260926 20:23:14.794826 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.794532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42171 (local address 127.0.0.1:51530)
0926 20:23:14.794712 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.794760 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260926 20:23:14.794946 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.252:42171: Network error: Client connection negotiation failed: client connection to 127.0.116.252:42171: connect: Connection refused (error 111)
I20260926 20:23:14.795837 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.795643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42171 (local address 127.0.0.1:51542)
0926 20:23:14.795741 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.795787 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 20:23:14.796039 13937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.252:42171: Network error: Client connection negotiation failed: client connection to 127.0.116.252:42171: connect: Connection refused (error 111)
I20260926 20:23:14.826149 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } attempt: 1
I20260926 20:23:14.827069 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.826747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42171 (local address 127.0.0.1:51552)
0926 20:23:14.826914 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.826986 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 20:23:14.827939 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.827707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42171 (local address 127.0.0.1:51556)
0926 20:23:14.827819 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.827875 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 20:23:14.828114 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.252:42171: Network error: Client connection negotiation failed: client connection to 127.0.116.252:42171: connect: Connection refused (error 111)
I20260926 20:23:14.845268 13937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } attempt: 1
I20260926 20:23:14.846194 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.845862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42171 (local address 127.0.0.1:51562)
0926 20:23:14.846033 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.846100 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 20:23:14.846954 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.846739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42171 (local address 127.0.0.1:51578)
0926 20:23:14.846848 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.846901 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 20:23:14.847076 13937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.252:42171: Network error: Client connection negotiation failed: client connection to 127.0.116.252:42171: connect: Connection refused (error 111)
I20260926 20:23:14.903291 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } attempt: 2
I20260926 20:23:14.904156 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.903865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42171 (local address 127.0.0.1:51588)
0926 20:23:14.904007 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.904077 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 20:23:14.904553 13944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:14.904892 13944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:14.904966 13944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 20:23:14.905066 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.904784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42171 (local address 127.0.0.1:51604)
0926 20:23:14.904887 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.904964 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 20:23:14.905269 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.252:42171: Network error: Client connection negotiation failed: client connection to 127.0.116.252:42171: connect: Connection refused (error 111)
W20260926 20:23:14.909288 13944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:23:14.909369 13944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:14.909415 13944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:14.909451 13944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:23:14.909487 13944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:23:14.914969 13944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:44333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:43525,127.0.116.253:44509,127.0.116.252:42171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.252.keytab
--principal=kudu/127.0.116.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.252:42171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.116.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:14.915249 13937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } attempt: 2
I20260926 20:23:14.916019 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.915745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42171 (local address 127.0.0.1:51610)
0926 20:23:14.915880 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.915948 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 20:23:14.916175 13944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:14.916661 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.916462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42171 (local address 127.0.0.1:51616)
0926 20:23:14.916561 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.916605 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 20:23:14.916786 13937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.252:42171: Network error: Client connection negotiation failed: client connection to 127.0.116.252:42171: connect: Connection refused (error 111)
I20260926 20:23:14.917285 13944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:14.925638 13953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:14.925658 13951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:14.925648 13950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:14.925959 13944 server_base.cc:1034] running on GCE node
I20260926 20:23:14.926545 13944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:14.927811 13944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:14.929003 13944 hybrid_clock.cc:648] HybridClock initialized: now 1790454194928982 us; error 29 us; skew 500 ppm
Sep 26 20:23:14 dist-test-slave-qsm9 krb5kdc[13781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:14.931959 13944 init.cc:377] Logged in from keytab as kudu/127.0.116.252@KRBTEST.COM (short username kudu)
I20260926 20:23:14.933020 13944 webserver.cc:460] Webserver started at http://127.0.116.252:44743/ using document root <none> and password file <none>
I20260926 20:23:14.933636 13944 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:14.933733 13944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:14.933979 13944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:23:14.936246 13944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-2/data/instance:
uuid: "ba98146659c1467e9d7e4d1239729e6c"
format_stamp: "Formatted at 2026-09-26 20:23:14 on dist-test-slave-qsm9"
server_key: "69340d8d9c53a309f3b49898354480e0"
server_key_iv: "578fe883952b744d2962b42d42644a98"
server_key_version: "encryptionkey@0"
I20260926 20:23:14.936860 13944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-2/wal/instance:
uuid: "ba98146659c1467e9d7e4d1239729e6c"
format_stamp: "Formatted at 2026-09-26 20:23:14 on dist-test-slave-qsm9"
server_key: "69340d8d9c53a309f3b49898354480e0"
server_key_iv: "578fe883952b744d2962b42d42644a98"
server_key_version: "encryptionkey@0"
I20260926 20:23:14.940749 13944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:23:14.943302 13960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:14.944729 13944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:23:14.944880 13944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-2/wal
uuid: "ba98146659c1467e9d7e4d1239729e6c"
format_stamp: "Formatted at 2026-09-26 20:23:14 on dist-test-slave-qsm9"
server_key: "69340d8d9c53a309f3b49898354480e0"
server_key_iv: "578fe883952b744d2962b42d42644a98"
server_key_version: "encryptionkey@0"
I20260926 20:23:14.945001 13944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:14.972851 13944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:14.976282 13944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:14.976512 13944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:14.984506 13944 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.252:42171
I20260926 20:23:14.984526 14012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.252:42171 every 8 connection(s)
I20260926 20:23:14.985556 13944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-2/data/info.pb
I20260926 20:23:14.986110 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 13944
I20260926 20:23:14.986254 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790454033974695-467-0/minicluster-data/master-2/wal/instance
I20260926 20:23:14.986531 467 external_mini_cluster.cc:1428] Setting key 431e27a7b6798923d99eb2b21f6eaaca
I20260926 20:23:14.988847 14013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:23:14.992718 14013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43525 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:23:14 dist-test-slave-qsm9 krb5kdc[13781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.252@KRBTEST.COM
Sep 26 20:23:14 dist-test-slave-qsm9 krb5kdc[13781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.252@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:23:15.002612 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.988005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.0.116.252:42171)
0926 20:23:14.988450 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:14.988465 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:14.988503 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:23:14.989290 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:14.989310 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:14.989661 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:14.990021 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:14.990863 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:14.991926 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:14.992819 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:14.993366 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:14.993541 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:14.996983 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:14.996989 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:14.996999 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:14.997027 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:14.999276 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:14.999857 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:14.999863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:14.999871 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:15.000057 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:15.000417 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:15.000424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:15.000428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:15.000911 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:15.001096 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:15.001422 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 20:23:15.001763 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":134,"threads_started":1}
I20260926 20:23:15.004437 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } attempt: 3
I20260926 20:23:15.005829 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.994779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41482 (local address 127.0.116.254:43525)
0926 20:23:14.994916 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:14.994919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:14.995298 (+ 379us) server_negotiation.cc:407] Connection header received
0926 20:23:14.995479 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:14.995488 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:14.995581 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:14.995669 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:14.996684 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:14.997172 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:14.998566 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:14.998823 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:14.998917 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:15.001768 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:15.001771 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:15.001773 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:15.001798 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:15.003290 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:15.003870 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:15.003873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:15.003874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:15.003961 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:15.004220 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:15.004222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:15.004223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:15.004463 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:15.004575 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:15.005462 (+ 887us) server_negotiation.cc:299] Negotiation successful
0926 20:23:15.005660 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:23:15.006583 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:14.994662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:43525 (local address 127.0.0.1:41482)
0926 20:23:14.995167 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:14.995208 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:14.995364 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:14.995759 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:14.995765 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:14.996298 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:14.996509 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:14.996522 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:14.997305 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:14.997312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:14.998388 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:14.998404 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:14.998950 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:14.998954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:14.999101 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:14.999811 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:14.999835 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:15.001558 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:15.003401 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:15.003411 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:15.003417 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:15.003761 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:15.004051 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:15.004054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:15.004056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:15.004147 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:15.004601 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:15.004609 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:15.005220 (+ 611us) client_negotiation.cc:769] Sending connection context
0926 20:23:15.006257 (+ 1037us) client_negotiation.cc:240] Negotiation successful
0926 20:23:15.006442 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":172,"threads_started":1}
I20260926 20:23:15.007797 14013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:23:15.007942 13937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } attempt: 3
Sep 26 20:23:15 dist-test-slave-qsm9 krb5kdc[13781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for kudu/127.0.116.252@KRBTEST.COM
Sep 26 20:23:15 dist-test-slave-qsm9 krb5kdc[13781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
Sep 26 20:23:15 dist-test-slave-qsm9 krb5kdc[13781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.253@KRBTEST.COM for kudu/127.0.116.252@KRBTEST.COM
I20260926 20:23:15.015892 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:15.004898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42171 (local address 127.0.0.1:51624)
0926 20:23:15.005079 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:15.005097 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:15.005182 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:15.005436 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:15.005439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:15.005985 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:15.006107 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:15.006114 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.007191 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.007197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:15.007954 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:15.007967 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.008652 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.008655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:15.008767 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:15.009293 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:15.009314 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:15.011136 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:15.013968 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:15.013973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:15.013976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:15.014295 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:15.014668 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:15.014672 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:15.014674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:15.014778 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:15.015246 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:15.015249 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:15.015352 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 20:23:15.015447 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 20:23:15.015625 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":3}
Sep 26 20:23:15 dist-test-slave-qsm9 krb5kdc[13781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.252@KRBTEST.COM for kudu/127.0.116.253@KRBTEST.COM
I20260926 20:23:15.016613 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:15.005065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51624 (local address 127.0.116.252:42171)
0926 20:23:15.005190 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:15.005195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:15.005209 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:23:15.005272 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:15.005275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:15.005314 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:15.005644 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:15.006213 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.007082 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.008085 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.008505 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.009006 (+ 501us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:15.011318 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:15.011322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:15.011325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:15.011351 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:15.013220 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:15.014412 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:15.014415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:15.014417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:15.014500 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:15.014904 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:15.014907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:15.014909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:15.015122 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:15.015847 (+ 725us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:15.015932 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 20:23:15.016378 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:23:15.017589 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:15.005435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41484 (local address 127.0.116.254:43525)
0926 20:23:15.005719 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:15.005739 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:15.005755 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:23:15.005798 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:15.005803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:15.005842 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:15.005990 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:15.006743 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.007651 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.008793 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.009188 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.009718 (+ 530us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:15.012545 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:15.012551 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:15.012554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:15.012589 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:15.014610 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:15.015179 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:15.015183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:15.015185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:15.015299 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:15.015740 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:15.015744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:15.015747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:15.016251 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:15.016485 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:15.017217 (+ 732us) server_negotiation.cc:299] Negotiation successful
0926 20:23:15.017356 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":209,"thread_start_us":86,"threads_started":1}
I20260926 20:23:15.020165 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:15.008360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42171 (local address 127.0.0.1:51630)
0926 20:23:15.008461 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:15.008477 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:15.008557 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:15.009137 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:15.009143 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:15.009358 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:15.009467 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:15.009475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.011084 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.011087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:15.011893 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:15.011917 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.012674 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.012678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:15.012764 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:15.013359 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:15.013374 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:15.015108 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:15.018147 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:15.018150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:15.018153 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:15.018582 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:15.018947 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:15.018951 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:15.018953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:15.019052 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:15.019574 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:15.019577 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:15.019695 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 20:23:15.019874 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 20:23:15.019991 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"spinlock_wait_cycles":9472}
I20260926 20:23:15.020707 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:15.009373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55476 (local address 127.0.116.253:44509)
0926 20:23:15.009467 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:15.009471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:15.009960 (+ 489us) server_negotiation.cc:407] Connection header received
0926 20:23:15.010162 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:15.010168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:15.010218 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:15.010266 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:15.010787 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.011541 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.013566 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.013977 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.014094 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:15.016589 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:15.016593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:15.016596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:15.016623 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:15.018829 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:15.019338 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:15.019341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:15.019343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:15.019420 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:15.019778 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:15.019782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:15.019784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:15.020003 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:15.020108 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:15.020435 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 20:23:15.020550 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:23:15.020761 14019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:15.008580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51630 (local address 127.0.116.252:42171)
0926 20:23:15.008888 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:15.008896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:15.008908 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:23:15.008947 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:15.008952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:15.008998 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:15.009577 (+ 579us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:15.009672 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.010975 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.012181 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.012551 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.012961 (+ 410us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:15.015249 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:15.015252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:15.015255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:15.015277 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:15.018029 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:15.018724 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:15.018728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:15.018730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:15.018828 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:15.019182 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:15.019185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:15.019187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:15.019421 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:15.020353 (+ 932us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:15.020426 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 20:23:15.020548 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":116,"threads_started":1}
I20260926 20:23:15.021128 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:15.008897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:44509 (local address 127.0.0.1:55476)
0926 20:23:15.009608 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:15.009620 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:15.010040 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:15.010388 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:15.010390 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:15.010559 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:15.010652 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:15.010658 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.011917 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.011922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:15.013431 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:15.013444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.014232 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.014236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:15.014360 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:15.014815 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:15.014834 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:15.016433 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:15.018947 (+ 2514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:15.018951 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:15.018953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:15.019221 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:15.019581 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:15.019584 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:15.019586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:15.019671 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:15.020120 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:15.020123 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:15.020230 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 20:23:15.020901 (+ 671us) client_negotiation.cc:240] Negotiation successful
0926 20:23:15.021024 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657}
I20260926 20:23:15.021159 13862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42: Bootstrap starting.
I20260926 20:23:15.023682 13862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42: Neither blocks nor log segments found. Creating new log.
I20260926 20:23:15.024399 14013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:23:15.024619 13862 log.cc:824] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42: Log is configured to *not* fsync() on all Append() calls
I20260926 20:23:15.027225 13862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42: No bootstrap required, opened a new log
I20260926 20:23:15.027194 13937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2: Bootstrap starting.
Sep 26 20:23:15 dist-test-slave-qsm9 krb5kdc[13781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.253@KRBTEST.COM
I20260926 20:23:15.030067 13937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2: Neither blocks nor log segments found. Creating new log.
I20260926 20:23:15.031210 13937 log.cc:824] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:23:15 dist-test-slave-qsm9 krb5kdc[13781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.252@KRBTEST.COM for kudu/127.0.116.252@KRBTEST.COM
I20260926 20:23:15.032933 13862 raft_consensus.cc:348] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66a9817d6e641cca8fd1176b492fa42" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43525 } } peers { permanent_uuid: "c084f09d19f44d33acc3b7c03747f9e2" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } } peers { permanent_uuid: "ba98146659c1467e9d7e4d1239729e6c" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } }
I20260926 20:23:15.033177 13862 raft_consensus.cc:374] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:23:15.033234 13862 raft_consensus.cc:729] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66a9817d6e641cca8fd1176b492fa42, State: Initialized, Role: FOLLOWER
I20260926 20:23:15.033620 13937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2: No bootstrap required, opened a new log
I20260926 20:23:15.034047 13862 consensus_queue.cc:260] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66a9817d6e641cca8fd1176b492fa42" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43525 } } peers { permanent_uuid: "c084f09d19f44d33acc3b7c03747f9e2" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } } peers { permanent_uuid: "ba98146659c1467e9d7e4d1239729e6c" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } }
I20260926 20:23:15.035200 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:15.023610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55480 (local address 127.0.116.253:44509)
0926 20:23:15.023762 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:15.023769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:15.023788 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:23:15.023837 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:15.023843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:15.023899 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:15.023984 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:15.025017 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.025839 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.026700 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.027057 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.027157 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:15.030782 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:15.030787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:15.030790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:15.030820 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:15.032727 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:15.033425 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:15.033431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:15.033434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:15.033534 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:15.033927 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:15.033931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:15.033933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:15.034220 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:15.034394 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:15.034828 (+ 434us) server_negotiation.cc:299] Negotiation successful
0926 20:23:15.034973 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:23:15.035776 13862 sys_catalog.cc:564] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:23:15.037281 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:15.024938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:42171 (local address 127.0.0.1:51636)
0926 20:23:15.026781 (+ 1843us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:15.026796 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:15.026878 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:15.027215 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:15.027218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:15.027432 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:15.027577 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:15.027585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.028653 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.028659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:15.029597 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:15.029609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.030398 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.030404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:15.030491 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:15.031024 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:15.031043 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:15.032856 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:15.035536 (+ 2680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:15.035539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:15.035542 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:15.035825 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:15.036215 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:15.036218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:15.036220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:15.036316 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:15.036781 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:15.036784 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:15.036881 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 20:23:15.036978 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 20:23:15.037115 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1757}
I20260926 20:23:15.037933 13937 raft_consensus.cc:348] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66a9817d6e641cca8fd1176b492fa42" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43525 } } peers { permanent_uuid: "c084f09d19f44d33acc3b7c03747f9e2" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } } peers { permanent_uuid: "ba98146659c1467e9d7e4d1239729e6c" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } }
I20260926 20:23:15.038125 14019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:15.024970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51636 (local address 127.0.116.252:42171)
0926 20:23:15.025363 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:15.025368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:15.026952 (+ 1584us) server_negotiation.cc:407] Connection header received
0926 20:23:15.026979 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:15.026982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:15.027021 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:15.027088 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:15.027718 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.028507 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.029744 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.030137 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.030263 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:15.033000 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:15.033004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:15.033006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:15.033035 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:15.035167 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:15.036003 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:15.036006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:15.036009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:15.036100 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:15.036417 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:15.036420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:15.036422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:15.036628 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:15.036741 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:15.037833 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0926 20:23:15.037964 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314}
I20260926 20:23:15.038188 13937 raft_consensus.cc:374] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:23:15.038246 13937 raft_consensus.cc:729] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c084f09d19f44d33acc3b7c03747f9e2, State: Initialized, Role: FOLLOWER
I20260926 20:23:15.038771 13937 consensus_queue.cc:260] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66a9817d6e641cca8fd1176b492fa42" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43525 } } peers { permanent_uuid: "c084f09d19f44d33acc3b7c03747f9e2" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } } peers { permanent_uuid: "ba98146659c1467e9d7e4d1239729e6c" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } }
I20260926 20:23:15.039675 14034 sys_catalog.cc:455] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66a9817d6e641cca8fd1176b492fa42" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43525 } } peers { permanent_uuid: "c084f09d19f44d33acc3b7c03747f9e2" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } } peers { permanent_uuid: "ba98146659c1467e9d7e4d1239729e6c" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } } }
I20260926 20:23:15.039932 14034 sys_catalog.cc:458] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:23:15.040131 13937 sys_catalog.cc:564] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:23:15.040966 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 20:23:15.042747 14033 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:23:15.042826 14033 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:23:15.043366 14013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c: Bootstrap starting.
I20260926 20:23:15.045584 14013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c: Neither blocks nor log segments found. Creating new log.
I20260926 20:23:15.046320 14013 log.cc:824] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:23:15.043296 13794 master_runner.cc:186] Error getting master registration for 127.0.116.253:44509: OK, instance_id { permanent_uuid: "c084f09d19f44d33acc3b7c03747f9e2" instance_seqno: 1790454194752073 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:23:15.046494 13794 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:23:15.048494 14013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c: No bootstrap required, opened a new log
I20260926 20:23:15.048805 14022 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66a9817d6e641cca8fd1176b492fa42" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43525 } } peers { permanent_uuid: "c084f09d19f44d33acc3b7c03747f9e2" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } } peers { permanent_uuid: "ba98146659c1467e9d7e4d1239729e6c" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } } }
I20260926 20:23:15.049011 14022 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:23:15.053093 14013 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66a9817d6e641cca8fd1176b492fa42" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43525 } } peers { permanent_uuid: "c084f09d19f44d33acc3b7c03747f9e2" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } } peers { permanent_uuid: "ba98146659c1467e9d7e4d1239729e6c" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } }
I20260926 20:23:15.053398 14013 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:23:15.053467 14013 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba98146659c1467e9d7e4d1239729e6c, State: Initialized, Role: FOLLOWER
W20260926 20:23:15.058663 14054 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:23:15.058979 14054 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:23:15.059654 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:15.043242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55496 (local address 127.0.116.253:44509)
0926 20:23:15.047307 (+ 4065us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:15.047312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:15.047339 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:23:15.047386 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:15.047388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:15.047429 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:15.047575 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:15.048465 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.049331 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.050474 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.050857 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.051398 (+ 541us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:15.052881 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:15.052886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:15.052890 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:15.052920 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:15.054801 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:15.056306 (+ 1505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:15.056310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:15.056312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:15.056406 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:15.057122 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:15.057126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:15.057127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:15.057369 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:15.059247 (+ 1878us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:15.059331 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 20:23:15.059452 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3987}
I20260926 20:23:15.060268 14019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:15.047164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51650 (local address 127.0.116.252:42171)
0926 20:23:15.047281 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:15.047286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:15.047855 (+ 569us) server_negotiation.cc:407] Connection header received
0926 20:23:15.047999 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:15.048003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:15.048053 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:15.048136 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:15.049510 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.050198 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.051425 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.051847 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.051938 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:15.053842 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:15.053846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:15.053849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:15.053883 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:15.056145 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:15.057505 (+ 1360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:15.057509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:15.057511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:15.057605 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:15.057939 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:15.057943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:15.057945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:15.058194 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:15.058337 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:15.059917 (+ 1580us) server_negotiation.cc:299] Negotiation successful
0926 20:23:15.060061 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:23:15.060055 14013 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66a9817d6e641cca8fd1176b492fa42" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43525 } } peers { permanent_uuid: "c084f09d19f44d33acc3b7c03747f9e2" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } } peers { permanent_uuid: "ba98146659c1467e9d7e4d1239729e6c" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } }
I20260926 20:23:15.061055 14056 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66a9817d6e641cca8fd1176b492fa42" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43525 } } peers { permanent_uuid: "c084f09d19f44d33acc3b7c03747f9e2" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } } peers { permanent_uuid: "ba98146659c1467e9d7e4d1239729e6c" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } } }
I20260926 20:23:15.061283 14056 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:23:15.062004 14013 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:23:15.062109 13869 master_runner.cc:186] Error getting master registration for 127.0.116.252:42171: OK, instance_id { permanent_uuid: "ba98146659c1467e9d7e4d1239729e6c" instance_seqno: 1790454194982276 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:23:15.065773 13869 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:23:15.070219 13944 master_runner.cc:263] No leader master found from master ba98146659c1467e9d7e4d1239729e6c
I20260926 20:23:15.070298 13944 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 20:23:15.070864 14067 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:23:15.071061 14067 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:23:15.072979 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:15.048353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41486 (local address 127.0.116.254:43525)
0926 20:23:15.048486 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:15.048491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:15.050685 (+ 2194us) server_negotiation.cc:407] Connection header received
0926 20:23:15.050870 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:15.050874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:15.050921 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:15.050969 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:15.052282 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.053584 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.066303 (+ 12719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:15.066734 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:15.066815 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:15.068894 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:15.068899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:15.068902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:15.068931 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:15.070839 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:15.071406 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:15.071410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:15.071412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:15.071549 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:15.071933 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:15.071937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:15.071939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:15.072195 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:15.072344 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:15.072679 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 20:23:15.072815 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 20:23:15.228155 13966 tablet.cc:2369] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:23:15.248833 13890 tablet.cc:2369] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:23:15.280797 13815 tablet.cc:2369] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:23:15.453109 14022 raft_consensus.cc:482] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:23:15.453397 14022 raft_consensus.cc:504] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66a9817d6e641cca8fd1176b492fa42" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43525 } } peers { permanent_uuid: "c084f09d19f44d33acc3b7c03747f9e2" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } } peers { permanent_uuid: "ba98146659c1467e9d7e4d1239729e6c" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } }
I20260926 20:23:15.454845 14022 leader_election.cc:288] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c084f09d19f44d33acc3b7c03747f9e2 (127.0.116.253:44509), ba98146659c1467e9d7e4d1239729e6c (127.0.116.252:42171)
I20260926 20:23:15.455622 13987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b66a9817d6e641cca8fd1176b492fa42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba98146659c1467e9d7e4d1239729e6c" is_pre_election: true
I20260926 20:23:15.456060 13987 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b66a9817d6e641cca8fd1176b492fa42 in term 0.
I20260926 20:23:15.456199 13911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b66a9817d6e641cca8fd1176b492fa42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c084f09d19f44d33acc3b7c03747f9e2" is_pre_election: true
I20260926 20:23:15.456552 13911 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b66a9817d6e641cca8fd1176b492fa42 in term 0.
I20260926 20:23:15.456598 13810 leader_election.cc:302] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b66a9817d6e641cca8fd1176b492fa42, ba98146659c1467e9d7e4d1239729e6c; no voters:
I20260926 20:23:15.456967 14022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:23:15.457058 14022 raft_consensus.cc:482] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:23:15.457110 14022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:23:15.458252 14022 raft_consensus.cc:504] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66a9817d6e641cca8fd1176b492fa42" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43525 } } peers { permanent_uuid: "c084f09d19f44d33acc3b7c03747f9e2" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } } peers { permanent_uuid: "ba98146659c1467e9d7e4d1239729e6c" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } }
I20260926 20:23:15.458600 14022 leader_election.cc:288] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [CANDIDATE]: Term 1 election: Requested vote from peers c084f09d19f44d33acc3b7c03747f9e2 (127.0.116.253:44509), ba98146659c1467e9d7e4d1239729e6c (127.0.116.252:42171)
I20260926 20:23:15.458971 13987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b66a9817d6e641cca8fd1176b492fa42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba98146659c1467e9d7e4d1239729e6c"
I20260926 20:23:15.458945 13911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b66a9817d6e641cca8fd1176b492fa42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c084f09d19f44d33acc3b7c03747f9e2"
I20260926 20:23:15.459100 13987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:23:15.459116 13911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:23:15.460291 13987 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b66a9817d6e641cca8fd1176b492fa42 in term 1.
I20260926 20:23:15.460320 13911 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b66a9817d6e641cca8fd1176b492fa42 in term 1.
I20260926 20:23:15.460601 13810 leader_election.cc:302] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b66a9817d6e641cca8fd1176b492fa42, ba98146659c1467e9d7e4d1239729e6c; no voters:
I20260926 20:23:15.460783 14022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:23:15.461014 14022 raft_consensus.cc:686] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [term 1 LEADER]: Becoming Leader. State: Replica: b66a9817d6e641cca8fd1176b492fa42, State: Running, Role: LEADER
I20260926 20:23:15.461403 14022 consensus_queue.cc:237] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66a9817d6e641cca8fd1176b492fa42" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43525 } } peers { permanent_uuid: "c084f09d19f44d33acc3b7c03747f9e2" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } } peers { permanent_uuid: "ba98146659c1467e9d7e4d1239729e6c" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } }
I20260926 20:23:15.463840 14069 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b66a9817d6e641cca8fd1176b492fa42. Latest consensus state: current_term: 1 leader_uuid: "b66a9817d6e641cca8fd1176b492fa42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66a9817d6e641cca8fd1176b492fa42" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43525 } } peers { permanent_uuid: "c084f09d19f44d33acc3b7c03747f9e2" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } } peers { permanent_uuid: "ba98146659c1467e9d7e4d1239729e6c" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } } }
I20260926 20:23:15.464015 14069 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [sys.catalog]: This master's current role is: LEADER
I20260926 20:23:15.464425 14071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:23:15.465245 14071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:23:15.470247 13987 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c [term 1 FOLLOWER]: Refusing update from remote peer b66a9817d6e641cca8fd1176b492fa42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:23:15.470387 13911 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2 [term 1 FOLLOWER]: Refusing update from remote peer b66a9817d6e641cca8fd1176b492fa42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:23:15.470957 14069 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba98146659c1467e9d7e4d1239729e6c" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:23:15.471122 14022 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c084f09d19f44d33acc3b7c03747f9e2" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:23:15.474810 14034 sys_catalog.cc:455] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b66a9817d6e641cca8fd1176b492fa42. Latest consensus state: current_term: 1 leader_uuid: "b66a9817d6e641cca8fd1176b492fa42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66a9817d6e641cca8fd1176b492fa42" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43525 } } peers { permanent_uuid: "c084f09d19f44d33acc3b7c03747f9e2" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } } peers { permanent_uuid: "ba98146659c1467e9d7e4d1239729e6c" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } } }
I20260926 20:23:15.474949 14034 sys_catalog.cc:458] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:23:15.474887 14056 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b66a9817d6e641cca8fd1176b492fa42. Latest consensus state: current_term: 1 leader_uuid: "b66a9817d6e641cca8fd1176b492fa42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66a9817d6e641cca8fd1176b492fa42" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43525 } } peers { permanent_uuid: "c084f09d19f44d33acc3b7c03747f9e2" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } } peers { permanent_uuid: "ba98146659c1467e9d7e4d1239729e6c" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } } }
I20260926 20:23:15.475005 14056 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:23:15.476969 14022 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b66a9817d6e641cca8fd1176b492fa42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66a9817d6e641cca8fd1176b492fa42" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43525 } } peers { permanent_uuid: "c084f09d19f44d33acc3b7c03747f9e2" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } } peers { permanent_uuid: "ba98146659c1467e9d7e4d1239729e6c" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } } }
I20260926 20:23:15.477102 14022 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [sys.catalog]: This master's current role is: LEADER
I20260926 20:23:15.477528 14022 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b66a9817d6e641cca8fd1176b492fa42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66a9817d6e641cca8fd1176b492fa42" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43525 } } peers { permanent_uuid: "c084f09d19f44d33acc3b7c03747f9e2" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } } peers { permanent_uuid: "ba98146659c1467e9d7e4d1239729e6c" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } } }
I20260926 20:23:15.477631 14022 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42 [sys.catalog]: This master's current role is: LEADER
I20260926 20:23:15.477569 14034 sys_catalog.cc:455] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b66a9817d6e641cca8fd1176b492fa42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66a9817d6e641cca8fd1176b492fa42" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43525 } } peers { permanent_uuid: "c084f09d19f44d33acc3b7c03747f9e2" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } } peers { permanent_uuid: "ba98146659c1467e9d7e4d1239729e6c" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } } }
I20260926 20:23:15.477700 14034 sys_catalog.cc:458] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:23:15.478619 14071 catalog_manager.cc:1347] Generated new cluster ID: 27da6c4201cf4e8e92ee6084710bb7c2
I20260926 20:23:15.478704 14071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:23:15.479067 14056 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b66a9817d6e641cca8fd1176b492fa42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b66a9817d6e641cca8fd1176b492fa42" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 43525 } } peers { permanent_uuid: "c084f09d19f44d33acc3b7c03747f9e2" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 44509 } } peers { permanent_uuid: "ba98146659c1467e9d7e4d1239729e6c" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 42171 } } }
I20260926 20:23:15.479206 14056 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:23:15.479391 14073 mvcc.cc:205] Tried to move back new op lower bound from 7333700384639213568 to 7333700384613146624. Current Snapshot: MvccSnapshot[applied={T|T < 7333700384639213568}]
I20260926 20:23:15.493173 14071 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:23:15.494331 14071 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:23:15.505533 14071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b66a9817d6e641cca8fd1176b492fa42: Generated new TSK 0
I20260926 20:23:15.506460 14071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:23:16.060714 14054 catalog_manager.cc:1259] Loaded cluster ID: 27da6c4201cf4e8e92ee6084710bb7c2
I20260926 20:23:16.060813 14054 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2: loading cluster ID for follower catalog manager: success
I20260926 20:23:16.063308 14054 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2: acquiring CA information for follower catalog manager: success
I20260926 20:23:16.064715 14054 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c084f09d19f44d33acc3b7c03747f9e2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:23:16.072510 14067 catalog_manager.cc:1259] Loaded cluster ID: 27da6c4201cf4e8e92ee6084710bb7c2
I20260926 20:23:16.072589 14067 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c: loading cluster ID for follower catalog manager: success
I20260926 20:23:16.074229 14067 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c: acquiring CA information for follower catalog manager: success
I20260926 20:23:16.075337 14067 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ba98146659c1467e9d7e4d1239729e6c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:23:16.578341 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 13794
I20260926 20:23:16.584617 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 13869
I20260926 20:23:16.590368 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 13944
2026-09-26T20:23:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3423 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:23:16 dist-test-slave-qsm9 krb5kdc[14087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:23:16 dist-test-slave-qsm9 krb5kdc[14087](info): set up 2 sockets
Sep 26 20:23:16 dist-test-slave-qsm9 krb5kdc[14087](info): commencing operation
krb5kdc: starting...
W20260926 20:23:17.712109 467 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:23:17 dist-test-slave-qsm9 krb5kdc[14087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:23:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:23:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:23:17.839056 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:45705
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:34017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:45705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:23:17.950565 14100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:17.950905 14100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:17.950989 14100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:17.955037 14100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:23:17.955129 14100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:17.955168 14100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:17.955202 14100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:23:17.955248 14100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:23:17.960589 14100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:34017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:45705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:45705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:17.961742 14100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:17.962895 14100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:17.971458 14108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:17.971436 14106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:17.971444 14100 server_base.cc:1034] running on GCE node
W20260926 20:23:17.971426 14105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:17.972323 14100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:17.973428 14100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:17.974586 14100 hybrid_clock.cc:648] HybridClock initialized: now 1790454197974572 us; error 28 us; skew 500 ppm
Sep 26 20:23:17 dist-test-slave-qsm9 krb5kdc[14087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:17.977363 14100 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:23:17.978268 14100 webserver.cc:460] Webserver started at http://127.0.116.254:34375/ using document root <none> and password file <none>
I20260926 20:23:17.978822 14100 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:17.978884 14100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:17.979101 14100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:23:17.981165 14100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "2334b14ae18b4c0a85dbd4cea7c01465"
format_stamp: "Formatted at 2026-09-26 20:23:17 on dist-test-slave-qsm9"
server_key: "db7ba546803d11dd4745a02ca9b68c35"
server_key_iv: "ddb90b58ffa715015034b45848114983"
server_key_version: "encryptionkey@0"
I20260926 20:23:17.981727 14100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "2334b14ae18b4c0a85dbd4cea7c01465"
format_stamp: "Formatted at 2026-09-26 20:23:17 on dist-test-slave-qsm9"
server_key: "db7ba546803d11dd4745a02ca9b68c35"
server_key_iv: "ddb90b58ffa715015034b45848114983"
server_key_version: "encryptionkey@0"
I20260926 20:23:17.985740 14100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 20:23:17.988193 14115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:17.989472 14100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 20:23:17.989698 14100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "2334b14ae18b4c0a85dbd4cea7c01465"
format_stamp: "Formatted at 2026-09-26 20:23:17 on dist-test-slave-qsm9"
server_key: "db7ba546803d11dd4745a02ca9b68c35"
server_key_iv: "ddb90b58ffa715015034b45848114983"
server_key_version: "encryptionkey@0"
I20260926 20:23:17.989815 14100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:18.000339 14100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:18.003335 14100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:18.003558 14100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:18.010665 14100 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:45705
I20260926 20:23:18.010695 14167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:45705 every 8 connection(s)
I20260926 20:23:18.011749 14100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:23:18.014631 14168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:23:18.015559 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 14100
I20260926 20:23:18.015661 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:23:18.016007 467 external_mini_cluster.cc:1428] Setting key f1518f6caa173bf76d6f8a06839ca61f
I20260926 20:23:18.021559 14168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465: Bootstrap starting.
I20260926 20:23:18.024290 14168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465: Neither blocks nor log segments found. Creating new log.
I20260926 20:23:18.025061 14168 log.cc:824] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:23:18 dist-test-slave-qsm9 krb5kdc[14087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:23:18.027086 14168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465: No bootstrap required, opened a new log
I20260926 20:23:18.031361 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:18.017374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58196 (local address 127.0.116.254:45705)
0926 20:23:18.017810 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:18.017829 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:18.017875 (+ 46us) server_negotiation.cc:407] Connection header received
0926 20:23:18.018723 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:18.018746 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:18.019169 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:18.019593 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:18.020304 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:18.021770 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:18.022682 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:18.023359 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:18.023576 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:18.026358 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:18.026364 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:18.026378 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:18.026406 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:18.028305 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:18.028843 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:18.028848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:18.028853 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:18.028968 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:18.029320 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:18.029324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:18.029325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:18.029779 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:18.029935 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:18.030234 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 20:23:18.030561 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":115,"threads_started":1}
I20260926 20:23:18.031384 14168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2334b14ae18b4c0a85dbd4cea7c01465" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45705 } }
I20260926 20:23:18.031728 14168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:23:18.031802 14168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2334b14ae18b4c0a85dbd4cea7c01465, State: Initialized, Role: FOLLOWER
I20260926 20:23:18.032281 14168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2334b14ae18b4c0a85dbd4cea7c01465" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45705 } }
I20260926 20:23:18.032430 14168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:23:18.032528 14168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:23:18.032621 14168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:23:18.033696 14168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2334b14ae18b4c0a85dbd4cea7c01465" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45705 } }
I20260926 20:23:18.034056 14168 leader_election.cc:302] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2334b14ae18b4c0a85dbd4cea7c01465; no voters:
I20260926 20:23:18.034394 14168 leader_election.cc:288] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:23:18.034510 14173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:23:18.034735 14173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465 [term 1 LEADER]: Becoming Leader. State: Replica: 2334b14ae18b4c0a85dbd4cea7c01465, State: Running, Role: LEADER
I20260926 20:23:18.034797 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:23:18.035133 14173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2334b14ae18b4c0a85dbd4cea7c01465" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45705 } }
I20260926 20:23:18.035863 14168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:23:18.036459 14175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2334b14ae18b4c0a85dbd4cea7c01465. Latest consensus state: current_term: 1 leader_uuid: "2334b14ae18b4c0a85dbd4cea7c01465" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2334b14ae18b4c0a85dbd4cea7c01465" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45705 } } }
I20260926 20:23:18.036598 14175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465 [sys.catalog]: This master's current role is: LEADER
I20260926 20:23:18.037003 14178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:23:18.037297 14174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2334b14ae18b4c0a85dbd4cea7c01465" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2334b14ae18b4c0a85dbd4cea7c01465" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45705 } } }
I20260926 20:23:18.037374 14174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465 [sys.catalog]: This master's current role is: LEADER
I20260926 20:23:18.040357 14178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:23:18.046082 14178 catalog_manager.cc:1347] Generated new cluster ID: a030fc064ad7436fbb7439c69bb1a143
I20260926 20:23:18.046157 14178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:23:18.051345 14178 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:23:18.052559 14178 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:23:18.068109 14178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2334b14ae18b4c0a85dbd4cea7c01465: Generated new TSK 0
I20260926 20:23:18.069042 14178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:23:18.091048 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 14100
2026-09-26T20:23:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1498 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:23:18 dist-test-slave-qsm9 krb5kdc[14195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:23:18 dist-test-slave-qsm9 krb5kdc[14195](info): set up 2 sockets
Sep 26 20:23:18 dist-test-slave-qsm9 krb5kdc[14195](info): commencing operation
krb5kdc: starting...
W20260926 20:23:19.132140 467 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:23:19 dist-test-slave-qsm9 krb5kdc[14195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:23:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:23:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:23:19.259234 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:46601
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:46601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:23:19.371536 14208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:19.371814 14208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:19.371856 14208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:19.376008 14208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:23:19.376070 14208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:19.376089 14208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:19.376106 14208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:23:19.376148 14208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:23:19.381592 14208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:46601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:46601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:19.382742 14208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:19.383872 14208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:19.392681 14213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:19.392674 14214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:19.392675 14216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:19.392827 14208 server_base.cc:1034] running on GCE node
I20260926 20:23:19.393787 14208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:19.395334 14208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:19.402753 14208 hybrid_clock.cc:648] HybridClock initialized: now 1790454199402711 us; error 72 us; skew 500 ppm
Sep 26 20:23:19 dist-test-slave-qsm9 krb5kdc[14195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:19.406767 14208 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:23:19.407888 14208 webserver.cc:460] Webserver started at http://127.0.116.254:43999/ using document root <none> and password file <none>
I20260926 20:23:19.408512 14208 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:19.408579 14208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:19.408798 14208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:23:19.411057 14208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "ff0e2eb98cb245028523b566380d48d7"
format_stamp: "Formatted at 2026-09-26 20:23:19 on dist-test-slave-qsm9"
server_key: "2f2fdd4a32dadad210078bb539135280"
server_key_iv: "437e56db8781c7ae4c2fcba32a58b57e"
server_key_version: "encryptionkey@0"
I20260926 20:23:19.411691 14208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "ff0e2eb98cb245028523b566380d48d7"
format_stamp: "Formatted at 2026-09-26 20:23:19 on dist-test-slave-qsm9"
server_key: "2f2fdd4a32dadad210078bb539135280"
server_key_iv: "437e56db8781c7ae4c2fcba32a58b57e"
server_key_version: "encryptionkey@0"
I20260926 20:23:19.415865 14208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:23:19.418471 14223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:19.419950 14208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:23:19.420104 14208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "ff0e2eb98cb245028523b566380d48d7"
format_stamp: "Formatted at 2026-09-26 20:23:19 on dist-test-slave-qsm9"
server_key: "2f2fdd4a32dadad210078bb539135280"
server_key_iv: "437e56db8781c7ae4c2fcba32a58b57e"
server_key_version: "encryptionkey@0"
I20260926 20:23:19.420224 14208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:19.432878 14208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:19.436084 14208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:19.436304 14208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:19.443938 14208 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:46601
I20260926 20:23:19.444237 14275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:46601 every 8 connection(s)
I20260926 20:23:19.445091 14208 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:23:19.448160 14276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:23:19.451480 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 14208
I20260926 20:23:19.451728 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:23:19.452023 467 external_mini_cluster.cc:1428] Setting key 0505f76018f0f0f83a2da19f133978aa
I20260926 20:23:19.454272 14276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7: Bootstrap starting.
I20260926 20:23:19.456522 14276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7: Neither blocks nor log segments found. Creating new log.
I20260926 20:23:19.457271 14276 log.cc:824] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7: Log is configured to *not* fsync() on all Append() calls
I20260926 20:23:19.459173 14276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7: No bootstrap required, opened a new log
Sep 26 20:23:19 dist-test-slave-qsm9 krb5kdc[14195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:23:19.463375 14276 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0e2eb98cb245028523b566380d48d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46601 } }
I20260926 20:23:19.463666 14276 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:23:19.463743 14276 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff0e2eb98cb245028523b566380d48d7, State: Initialized, Role: FOLLOWER
I20260926 20:23:19.464254 14276 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0e2eb98cb245028523b566380d48d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46601 } }
I20260926 20:23:19.464380 14276 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:23:19.464465 14276 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:23:19.464573 14276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:23:19.465519 14276 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0e2eb98cb245028523b566380d48d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46601 } }
I20260926 20:23:19.465884 14276 leader_election.cc:302] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff0e2eb98cb245028523b566380d48d7; no voters:
I20260926 20:23:19.466277 14276 leader_election.cc:288] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:23:19.466362 14281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:23:19.466540 14281 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 1 LEADER]: Becoming Leader. State: Replica: ff0e2eb98cb245028523b566380d48d7, State: Running, Role: LEADER
I20260926 20:23:19.466872 14281 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0e2eb98cb245028523b566380d48d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46601 } }
I20260926 20:23:19.467202 14279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:19.453503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42166 (local address 127.0.116.254:46601)
0926 20:23:19.453951 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:19.453966 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:19.454002 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:23:19.454708 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:19.454727 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:19.455068 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:19.455402 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:19.456263 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:19.457310 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:19.458189 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:19.458678 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:19.458846 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:19.461723 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:19.461729 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:19.461744 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:19.461776 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:19.463817 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:19.464301 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:19.464308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:19.464314 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:19.464482 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:19.464813 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:19.464816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:19.464818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:19.465249 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:19.465458 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:19.465757 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 20:23:19.466120 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":134,"threads_started":1}
I20260926 20:23:19.467666 14276 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:23:19.468613 14283 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff0e2eb98cb245028523b566380d48d7. Latest consensus state: current_term: 1 leader_uuid: "ff0e2eb98cb245028523b566380d48d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0e2eb98cb245028523b566380d48d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46601 } } }
I20260926 20:23:19.468737 14283 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [sys.catalog]: This master's current role is: LEADER
I20260926 20:23:19.469058 14282 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff0e2eb98cb245028523b566380d48d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0e2eb98cb245028523b566380d48d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46601 } } }
I20260926 20:23:19.469141 14282 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [sys.catalog]: This master's current role is: LEADER
I20260926 20:23:19.470217 14285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:23:19.472909 14285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:23:19.478752 14285 catalog_manager.cc:1347] Generated new cluster ID: 977edc0f6d874980b3c25b6b277af0fd
I20260926 20:23:19.478857 14285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:23:19.499166 14285 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:23:19.500283 14285 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:23:19.507189 14285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7: Generated new TSK 0
I20260926 20:23:19.508026 14285 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
I20260926 20:23:19.539381 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46601
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:23:19.653132 14302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:19.653416 14302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:19.653461 14302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:19.657689 14302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:19.657753 14302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:19.657850 14302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:23:19.663208 14302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:19.664346 14302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:19.665441 14302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:19.675280 14307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:19.675311 14310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:19.675602 14302 server_base.cc:1034] running on GCE node
W20260926 20:23:19.675709 14308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:19.676100 14302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:19.676702 14302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:19.677862 14302 hybrid_clock.cc:648] HybridClock initialized: now 1790454199677834 us; error 45 us; skew 500 ppm
Sep 26 20:23:19 dist-test-slave-qsm9 krb5kdc[14195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:19.680828 14302 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:23:19.681733 14302 webserver.cc:460] Webserver started at http://127.0.116.193:34077/ using document root <none> and password file <none>
I20260926 20:23:19.682265 14302 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:19.682329 14302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:19.682492 14302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:23:19.684563 14302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "a410b1638b794d4b9f5475aee9fed5e1"
format_stamp: "Formatted at 2026-09-26 20:23:19 on dist-test-slave-qsm9"
server_key: "c3d457adf56dacf4d7656ebcb94a0e42"
server_key_iv: "d2f83fbdc0d1eb4b3d4a8299d0e99ebc"
server_key_version: "encryptionkey@0"
I20260926 20:23:19.685081 14302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "a410b1638b794d4b9f5475aee9fed5e1"
format_stamp: "Formatted at 2026-09-26 20:23:19 on dist-test-slave-qsm9"
server_key: "c3d457adf56dacf4d7656ebcb94a0e42"
server_key_iv: "d2f83fbdc0d1eb4b3d4a8299d0e99ebc"
server_key_version: "encryptionkey@0"
I20260926 20:23:19.689071 14302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:23:19.691448 14317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:19.692685 14302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:23:19.692850 14302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "a410b1638b794d4b9f5475aee9fed5e1"
format_stamp: "Formatted at 2026-09-26 20:23:19 on dist-test-slave-qsm9"
server_key: "c3d457adf56dacf4d7656ebcb94a0e42"
server_key_iv: "d2f83fbdc0d1eb4b3d4a8299d0e99ebc"
server_key_version: "encryptionkey@0"
I20260926 20:23:19.692966 14302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:19.717162 14302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:19.720767 14302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:19.720993 14302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:19.721621 14302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:23:19.722541 14302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:23:19.722612 14302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:19.722688 14302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:23:19.722728 14302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:19.732174 14302 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:37055
I20260926 20:23:19.732199 14431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:37055 every 8 connection(s)
I20260926 20:23:19.733134 14302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:23:19.736471 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 14302
I20260926 20:23:19.736603 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:23:19.736888 467 external_mini_cluster.cc:1428] Setting key e9fe7d87df4786defd4f449693602468
Sep 26 20:23:19 dist-test-slave-qsm9 krb5kdc[14195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.116.194@KRBTEST.COM; defaulting to no policy
I20260926 20:23:19.750025 14279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:19.735204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:42691 (local address 127.0.116.254:46601)
0926 20:23:19.735390 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:19.735395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:19.736546 (+ 1151us) server_negotiation.cc:407] Connection header received
0926 20:23:19.737688 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:19.737692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:19.737749 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:19.737848 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:19.739267 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:19.740196 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:19.742144 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:19.742382 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:19.742462 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:19.745886 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:19.745891 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:19.745894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:19.745935 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:19.747901 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:19.748518 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:19.748522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:19.748523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:19.748606 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:19.748935 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:19.748938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:19.748939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:19.749172 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:19.749251 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:19.749773 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 20:23:19.749895 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.194@KRBTEST.COM" created.
I20260926 20:23:19.750996 14434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:19.735684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.193:42691)
0926 20:23:19.736384 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:19.736429 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:19.737435 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:19.738058 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:19.738067 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:19.738579 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:19.739098 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:19.739112 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:19.740358 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:19.740363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:19.741991 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:19.742008 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:19.742551 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:19.742557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:19.742811 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:19.743613 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:19.743638 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:19.745633 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:19.748067 (+ 2434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:19.748074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:19.748101 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:19.748410 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:19.748693 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:19.748695 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:19.748696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:19.748854 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:19.749266 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:19.749272 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:19.749555 (+ 283us) client_negotiation.cc:769] Sending connection context
0926 20:23:19.749731 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 20:23:19.750052 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":116,"threads_started":1}
I20260926 20:23:19.752472 14432 heartbeater.cc:347] Connected to a master server at 127.0.116.254:46601
I20260926 20:23:19.752741 14432 heartbeater.cc:464] Registering TS with master...
I20260926 20:23:19.753265 14432 heartbeater.cc:511] Master 127.0.116.254:46601 requested a full tablet report, sending...
I20260926 20:23:19.754858 14240 ts_manager.cc:194] Registered new tserver with Master: a410b1638b794d4b9f5475aee9fed5e1 (127.0.116.193:37055)
I20260926 20:23:19.756134 14240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:42691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
I20260926 20:23:19.768069 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46601
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:23:19.882265 14437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:19.882548 14437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:19.882592 14437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:19.886706 14437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:19.886770 14437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:19.886831 14437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:23:19.892534 14437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:19.893635 14437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:19.894708 14437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:19.904517 14442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:19.904503 14445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:19.904508 14443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:19.904870 14437 server_base.cc:1034] running on GCE node
I20260926 20:23:19.905290 14437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:19.905985 14437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:19.907202 14437 hybrid_clock.cc:648] HybridClock initialized: now 1790454199907178 us; error 32 us; skew 500 ppm
Sep 26 20:23:19 dist-test-slave-qsm9 krb5kdc[14195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:19.909991 14437 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:23:19.910950 14437 webserver.cc:460] Webserver started at http://127.0.116.194:33111/ using document root <none> and password file <none>
I20260926 20:23:19.911576 14437 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:19.911664 14437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:19.911872 14437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:23:19.913918 14437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "1baa9bc83c5e423198f49cef556af54e"
format_stamp: "Formatted at 2026-09-26 20:23:19 on dist-test-slave-qsm9"
server_key: "40d8ec69c428090b1d549185bc50694b"
server_key_iv: "f484f6529cc4173b2088ba8c5c45026b"
server_key_version: "encryptionkey@0"
I20260926 20:23:19.914503 14437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "1baa9bc83c5e423198f49cef556af54e"
format_stamp: "Formatted at 2026-09-26 20:23:19 on dist-test-slave-qsm9"
server_key: "40d8ec69c428090b1d549185bc50694b"
server_key_iv: "f484f6529cc4173b2088ba8c5c45026b"
server_key_version: "encryptionkey@0"
I20260926 20:23:19.918270 14437 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 20:23:19.920735 14452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:19.922034 14437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 20:23:19.922206 14437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "1baa9bc83c5e423198f49cef556af54e"
format_stamp: "Formatted at 2026-09-26 20:23:19 on dist-test-slave-qsm9"
server_key: "40d8ec69c428090b1d549185bc50694b"
server_key_iv: "f484f6529cc4173b2088ba8c5c45026b"
server_key_version: "encryptionkey@0"
I20260926 20:23:19.922328 14437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:19.938511 14437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:19.941614 14437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:19.941828 14437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:19.942451 14437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:23:19.943292 14437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:23:19.943361 14437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:19.943434 14437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:23:19.943475 14437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:19.952636 14437 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:37991
I20260926 20:23:19.952664 14565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:37991 every 8 connection(s)
I20260926 20:23:19.953614 14437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:23:19.955087 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 14437
I20260926 20:23:19.955219 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:23:19.955569 467 external_mini_cluster.cc:1428] Setting key 6af2c643ee022321377ebbaf967a4361
Sep 26 20:23:19 dist-test-slave-qsm9 krb5kdc[14195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.116.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.195@KRBTEST.COM" created.
I20260926 20:23:19.969660 14279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:19.955691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:49485 (local address 127.0.116.254:46601)
0926 20:23:19.955850 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:19.955855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:19.957102 (+ 1247us) server_negotiation.cc:407] Connection header received
0926 20:23:19.958051 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:19.958057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:19.958118 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:19.958210 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:19.959390 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:19.959992 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:19.961797 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:19.962058 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:19.962145 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:19.965202 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:19.965206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:19.965208 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:19.965233 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:19.966911 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:19.967688 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:19.967693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:19.967699 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:19.967800 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:19.968220 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:19.968233 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:19.968234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:19.968534 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:19.968624 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:19.969278 (+ 654us) server_negotiation.cc:299] Negotiation successful
0926 20:23:19.969478 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:23:19.970719 14568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:19.956174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.194:49485)
0926 20:23:19.956942 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:19.957002 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:19.957839 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:19.958378 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:19.958386 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:19.958755 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:19.959237 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:19.959250 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:19.960116 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:19.960122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:19.961646 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:19.961661 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:19.962227 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:19.962232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:19.962506 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:19.963200 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:19.963229 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:19.964989 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:19.967043 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:19.967050 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:19.967068 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:19.967566 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:19.967912 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:19.967915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:19.967917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:19.968078 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:19.968632 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:19.968638 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:19.968966 (+ 328us) client_negotiation.cc:769] Sending connection context
0926 20:23:19.969464 (+ 498us) client_negotiation.cc:240] Negotiation successful
0926 20:23:19.969802 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":118,"threads_started":1}
I20260926 20:23:19.971963 14566 heartbeater.cc:347] Connected to a master server at 127.0.116.254:46601
I20260926 20:23:19.972182 14566 heartbeater.cc:464] Registering TS with master...
I20260926 20:23:19.972636 14566 heartbeater.cc:511] Master 127.0.116.254:46601 requested a full tablet report, sending...
I20260926 20:23:19.973717 14240 ts_manager.cc:194] Registered new tserver with Master: 1baa9bc83c5e423198f49cef556af54e (127.0.116.194:37991)
I20260926 20:23:19.974610 14240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:49485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
I20260926 20:23:19.987434 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46601
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:23:20.104380 14571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:20.104710 14571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:20.104760 14571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:20.109411 14571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:20.109493 14571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:20.109582 14571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:23:20.115219 14571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:20.116453 14571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:20.117588 14571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:20.127990 14577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:20.127982 14576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:20.128239 14571 server_base.cc:1034] running on GCE node
W20260926 20:23:20.128008 14579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:20.128722 14571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:20.129391 14571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:20.130564 14571 hybrid_clock.cc:648] HybridClock initialized: now 1790454200130517 us; error 60 us; skew 500 ppm
Sep 26 20:23:20 dist-test-slave-qsm9 krb5kdc[14195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:20.133565 14571 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:23:20.134502 14571 webserver.cc:460] Webserver started at http://127.0.116.195:40359/ using document root <none> and password file <none>
I20260926 20:23:20.135064 14571 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:20.135123 14571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:20.135349 14571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:23:20.137455 14571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "f9aa02cf617040d78c5f86ce7f144961"
format_stamp: "Formatted at 2026-09-26 20:23:20 on dist-test-slave-qsm9"
server_key: "35dd14c426d0249f64d795ba41e0d4ea"
server_key_iv: "8a2e3bee538f3b0d9aa0772aa62177d2"
server_key_version: "encryptionkey@0"
I20260926 20:23:20.138008 14571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "f9aa02cf617040d78c5f86ce7f144961"
format_stamp: "Formatted at 2026-09-26 20:23:20 on dist-test-slave-qsm9"
server_key: "35dd14c426d0249f64d795ba41e0d4ea"
server_key_iv: "8a2e3bee538f3b0d9aa0772aa62177d2"
server_key_version: "encryptionkey@0"
I20260926 20:23:20.141772 14571 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 20:23:20.144268 14586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:20.145560 14571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:23:20.145653 14571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "f9aa02cf617040d78c5f86ce7f144961"
format_stamp: "Formatted at 2026-09-26 20:23:20 on dist-test-slave-qsm9"
server_key: "35dd14c426d0249f64d795ba41e0d4ea"
server_key_iv: "8a2e3bee538f3b0d9aa0772aa62177d2"
server_key_version: "encryptionkey@0"
I20260926 20:23:20.145740 14571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:20.159654 14571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:20.163257 14571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:20.163432 14571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:20.164069 14571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:23:20.164866 14571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:23:20.164919 14571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:20.164960 14571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:23:20.164981 14571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:20.174439 14571 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:39271
I20260926 20:23:20.174521 14699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:39271 every 8 connection(s)
I20260926 20:23:20.175386 14571 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:23:20.184870 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 14571
I20260926 20:23:20.185002 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:23:20.185395 467 external_mini_cluster.cc:1428] Setting key 1ff73eee0cfa0eb54efdbf906bcafec0
Sep 26 20:23:20 dist-test-slave-qsm9 krb5kdc[14195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:23:20.190425 14279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:20.177284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:46817 (local address 127.0.116.254:46601)
0926 20:23:20.177434 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:20.177438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:20.178293 (+ 855us) server_negotiation.cc:407] Connection header received
0926 20:23:20.179749 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:20.179753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:20.179823 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:20.179937 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:20.181216 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.181812 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.183081 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.183337 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.183458 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:20.186507 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:20.186512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:20.186515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:20.186544 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:20.188326 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:20.188861 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:20.188864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:20.188866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:20.188961 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:20.189275 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:20.189277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:20.189278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:20.189460 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:20.189534 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:20.190169 (+ 635us) server_negotiation.cc:299] Negotiation successful
0926 20:23:20.190294 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:23:20.191358 14702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:20.177619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.195:46817)
0926 20:23:20.178109 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:20.178154 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:20.179470 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:20.180128 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:20.180136 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:20.180538 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:20.181056 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:20.181071 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.181948 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.181952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:20.182938 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:20.182947 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.183543 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.183549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:20.183726 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:20.184349 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:20.184372 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:20.186327 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:20.188438 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:20.188444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:20.188459 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:20.188761 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:20.189048 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:20.189051 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:20.189052 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:20.189196 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:20.189545 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:20.189551 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:20.189902 (+ 351us) client_negotiation.cc:769] Sending connection context
0926 20:23:20.190088 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 20:23:20.190399 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":110,"threads_started":1}
I20260926 20:23:20.192622 14700 heartbeater.cc:347] Connected to a master server at 127.0.116.254:46601
I20260926 20:23:20.192904 14700 heartbeater.cc:464] Registering TS with master...
I20260926 20:23:20.193395 14700 heartbeater.cc:511] Master 127.0.116.254:46601 requested a full tablet report, sending...
I20260926 20:23:20.194487 14240 ts_manager.cc:194] Registered new tserver with Master: f9aa02cf617040d78c5f86ce7f144961 (127.0.116.195:39271)
I20260926 20:23:20.195097 14240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:46817
I20260926 20:23:20.199647 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:23:20.209327 14279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:20.201678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42174 (local address 127.0.116.254:46601)
0926 20:23:20.201813 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:20.201817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:20.201941 (+ 124us) server_negotiation.cc:407] Connection header received
0926 20:23:20.202007 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:20.202009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:20.202053 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:20.202131 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:20.202822 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.203407 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.204304 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.204623 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.204745 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:20.206155 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:20.206159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:20.206162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:20.206187 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:20.207836 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:20.208205 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:20.208208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:20.208209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:20.208282 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:20.208499 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:20.208502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:20.208503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:20.208756 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:20.208904 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:20.209098 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 20:23:20.209202 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:23:20.212481 14240 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:23:20.215842 14240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:23:20.231995 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:20.225909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34694 (local address 127.0.116.195:39271)
0926 20:23:20.226484 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:20.226494 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:20.226512 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:23:20.226572 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:20.226576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:20.226727 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:20.226893 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:23:20.227227 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.228502 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.230314 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.231209 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.231340 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:20.231540 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:20.231720 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 20:23:20.231834 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":128,"threads_started":1}
I20260926 20:23:20.232023 14713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:20.225719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:39271 (local address 127.0.0.1:34694)
0926 20:23:20.226132 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:20.226148 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:20.226230 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:20.226891 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:20.226898 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:20.226921 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:23:20.227102 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:20.227112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.228666 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.228671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:20.230195 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:20.230206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.231344 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.231348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:20.231477 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:20.231549 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 20:23:20.231681 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 20:23:20.231812 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":240,"thread_start_us":124,"threads_started":1}
I20260926 20:23:20.232028 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:20.225309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:37991 (local address 127.0.0.1:43366)
0926 20:23:20.225986 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:20.226002 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:20.226104 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:20.227060 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:20.227065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:20.227088 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:23:20.227213 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:20.227219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.228625 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.228628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:20.230161 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:20.230173 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.231590 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.231594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:20.231711 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:20.231736 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:23:20.231808 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 20:23:20.231876 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":110,"threads_started":1}
I20260926 20:23:20.232366 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:20.225720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43366 (local address 127.0.116.194:37991)
0926 20:23:20.226387 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:20.226401 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:20.226425 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:23:20.226506 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:20.226512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:20.226727 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:20.226893 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:23:20.227362 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.228502 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.230308 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.231246 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.231347 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:20.231528 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:20.232091 (+ 563us) server_negotiation.cc:299] Negotiation successful
0926 20:23:20.232221 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":160,"threads_started":1}
I20260926 20:23:20.232931 14711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:20.224948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:37055 (local address 127.0.0.1:38220)
0926 20:23:20.225337 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:20.225386 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:20.225513 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:20.227209 (+ 1696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:20.227213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:20.227237 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:23:20.227330 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:20.227335 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.228576 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.228583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:20.231053 (+ 2470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:20.231062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.232511 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.232517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:20.232632 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:20.232656 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:23:20.232710 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 20:23:20.232807 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":204,"thread_start_us":110,"threads_started":1}
I20260926 20:23:20.233529 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:20.225081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.0.116.193:37055)
0926 20:23:20.226550 (+ 1469us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:20.226569 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:20.226596 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:23:20.226750 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:20.226759 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:20.226966 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:20.227464 (+ 498us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:23:20.227676 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.228431 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.231194 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.232144 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.233104 (+ 960us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:20.233271 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:20.233325 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 20:23:20.233412 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1297,"thread_start_us":138,"threads_started":1}
I20260926 20:23:20.235699 14634 tablet_service.cc:1467] Processing CreateTablet for tablet 2472efdebbe6475eb3d0531c876e2a13 (DEFAULT_TABLE table=test-table [id=dd5c233419624ca9a1c210d616122c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:23:20.235687 14500 tablet_service.cc:1467] Processing CreateTablet for tablet 2472efdebbe6475eb3d0531c876e2a13 (DEFAULT_TABLE table=test-table [id=dd5c233419624ca9a1c210d616122c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:23:20.236088 14365 tablet_service.cc:1467] Processing CreateTablet for tablet 2472efdebbe6475eb3d0531c876e2a13 (DEFAULT_TABLE table=test-table [id=dd5c233419624ca9a1c210d616122c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:23:20.236835 14634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2472efdebbe6475eb3d0531c876e2a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:23:20.236835 14500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2472efdebbe6475eb3d0531c876e2a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:23:20.236961 14365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2472efdebbe6475eb3d0531c876e2a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:23:20.243607 14717 tablet_bootstrap.cc:492] T 2472efdebbe6475eb3d0531c876e2a13 P f9aa02cf617040d78c5f86ce7f144961: Bootstrap starting.
I20260926 20:23:20.244696 14718 tablet_bootstrap.cc:492] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e: Bootstrap starting.
I20260926 20:23:20.247118 14718 tablet_bootstrap.cc:654] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e: Neither blocks nor log segments found. Creating new log.
I20260926 20:23:20.247129 14717 tablet_bootstrap.cc:654] T 2472efdebbe6475eb3d0531c876e2a13 P f9aa02cf617040d78c5f86ce7f144961: Neither blocks nor log segments found. Creating new log.
I20260926 20:23:20.247712 14719 tablet_bootstrap.cc:492] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1: Bootstrap starting.
I20260926 20:23:20.248065 14718 log.cc:824] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:23:20.248076 14717 log.cc:824] T 2472efdebbe6475eb3d0531c876e2a13 P f9aa02cf617040d78c5f86ce7f144961: Log is configured to *not* fsync() on all Append() calls
I20260926 20:23:20.250303 14719 tablet_bootstrap.cc:654] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1: Neither blocks nor log segments found. Creating new log.
I20260926 20:23:20.250609 14718 tablet_bootstrap.cc:492] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e: No bootstrap required, opened a new log
I20260926 20:23:20.250864 14718 ts_tablet_manager.cc:1397] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:23:20.251737 14719 log.cc:824] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1: Log is configured to *not* fsync() on all Append() calls
I20260926 20:23:20.252923 14717 tablet_bootstrap.cc:492] T 2472efdebbe6475eb3d0531c876e2a13 P f9aa02cf617040d78c5f86ce7f144961: No bootstrap required, opened a new log
I20260926 20:23:20.253140 14717 ts_tablet_manager.cc:1397] T 2472efdebbe6475eb3d0531c876e2a13 P f9aa02cf617040d78c5f86ce7f144961: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260926 20:23:20.253850 14719 tablet_bootstrap.cc:492] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1: No bootstrap required, opened a new log
I20260926 20:23:20.254047 14719 ts_tablet_manager.cc:1397] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260926 20:23:20.255245 14718 raft_consensus.cc:348] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } }
I20260926 20:23:20.255600 14718 raft_consensus.cc:374] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:23:20.255661 14718 raft_consensus.cc:729] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1baa9bc83c5e423198f49cef556af54e, State: Initialized, Role: FOLLOWER
I20260926 20:23:20.256191 14718 consensus_queue.cc:260] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } }
I20260926 20:23:20.256245 14717 raft_consensus.cc:348] T 2472efdebbe6475eb3d0531c876e2a13 P f9aa02cf617040d78c5f86ce7f144961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } }
I20260926 20:23:20.256512 14717 raft_consensus.cc:374] T 2472efdebbe6475eb3d0531c876e2a13 P f9aa02cf617040d78c5f86ce7f144961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:23:20.256579 14717 raft_consensus.cc:729] T 2472efdebbe6475eb3d0531c876e2a13 P f9aa02cf617040d78c5f86ce7f144961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9aa02cf617040d78c5f86ce7f144961, State: Initialized, Role: FOLLOWER
I20260926 20:23:20.257059 14566 heartbeater.cc:503] Master 127.0.116.254:46601 was elected leader, sending a full tablet report...
I20260926 20:23:20.257014 14717 consensus_queue.cc:260] T 2472efdebbe6475eb3d0531c876e2a13 P f9aa02cf617040d78c5f86ce7f144961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } }
I20260926 20:23:20.257352 14718 ts_tablet_manager.cc:1428] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260926 20:23:20.258286 14700 heartbeater.cc:503] Master 127.0.116.254:46601 was elected leader, sending a full tablet report...
I20260926 20:23:20.258131 14719 raft_consensus.cc:348] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } }
I20260926 20:23:20.258395 14719 raft_consensus.cc:374] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:23:20.258445 14719 raft_consensus.cc:729] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a410b1638b794d4b9f5475aee9fed5e1, State: Initialized, Role: FOLLOWER
I20260926 20:23:20.258500 14717 ts_tablet_manager.cc:1428] T 2472efdebbe6475eb3d0531c876e2a13 P f9aa02cf617040d78c5f86ce7f144961: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 20:23:20.258908 14719 consensus_queue.cc:260] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } }
I20260926 20:23:20.259608 14432 heartbeater.cc:503] Master 127.0.116.254:46601 was elected leader, sending a full tablet report...
I20260926 20:23:20.260035 14719 ts_tablet_manager.cc:1428] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260926 20:23:20.382978 14723 raft_consensus.cc:482] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:23:20.383211 14723 raft_consensus.cc:504] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } }
I20260926 20:23:20.384470 14723 leader_election.cc:288] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a410b1638b794d4b9f5475aee9fed5e1 (127.0.116.193:37055), f9aa02cf617040d78c5f86ce7f144961 (127.0.116.195:39271)
I20260926 20:23:20.389039 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:20.384883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:35537 (local address 127.0.116.193:37055)
0926 20:23:20.385056 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:20.385062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:20.385085 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:23:20.385150 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:20.385154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:20.385245 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:20.385353 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:23:20.385673 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.386433 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.387847 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.388390 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.388527 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:20.388584 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:20.388793 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 20:23:20.388887 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:23:20.389068 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:20.385240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:34139 (local address 127.0.116.195:39271)
0926 20:23:20.385382 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:20.385387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:20.385567 (+ 180us) server_negotiation.cc:407] Connection header received
0926 20:23:20.385689 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:20.385691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:20.385729 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:20.385785 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:23:20.386117 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.386786 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.387968 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.388401 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.388541 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:20.388594 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:20.388878 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 20:23:20.388951 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:23:20.389055 14568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:20.384783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:37055 (local address 127.0.116.194:35537)
0926 20:23:20.384917 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:20.384933 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:20.385034 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:20.385383 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:20.385387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:20.385409 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:23:20.385517 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:20.385522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.386566 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.386571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:20.387695 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:20.387706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.388531 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.388535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:20.388660 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:20.388680 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 20:23:20.388786 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 20:23:20.388890 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 20:23:20.389072 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:20.385059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:39271 (local address 127.0.116.194:34139)
0926 20:23:20.385459 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:20.385481 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:20.385570 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:20.385822 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:20.385825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:20.385844 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:23:20.385962 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:20.385969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.386913 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.386916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:20.387850 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:20.387861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.388551 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.388554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:20.388662 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:20.388762 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 20:23:20.388849 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 20:23:20.388916 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":132,"threads_started":1}
I20260926 20:23:20.389807 14654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2472efdebbe6475eb3d0531c876e2a13" candidate_uuid: "1baa9bc83c5e423198f49cef556af54e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9aa02cf617040d78c5f86ce7f144961" is_pre_election: true
I20260926 20:23:20.389839 14385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2472efdebbe6475eb3d0531c876e2a13" candidate_uuid: "1baa9bc83c5e423198f49cef556af54e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a410b1638b794d4b9f5475aee9fed5e1" is_pre_election: true
I20260926 20:23:20.390232 14385 raft_consensus.cc:2454] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1baa9bc83c5e423198f49cef556af54e in term 0.
I20260926 20:23:20.390332 14654 raft_consensus.cc:2454] T 2472efdebbe6475eb3d0531c876e2a13 P f9aa02cf617040d78c5f86ce7f144961 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1baa9bc83c5e423198f49cef556af54e in term 0.
I20260926 20:23:20.390760 14454 leader_election.cc:302] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1baa9bc83c5e423198f49cef556af54e, a410b1638b794d4b9f5475aee9fed5e1; no voters:
I20260926 20:23:20.391031 14723 raft_consensus.cc:2781] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:23:20.391124 14723 raft_consensus.cc:482] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:23:20.391197 14723 raft_consensus.cc:3037] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:23:20.392391 14723 raft_consensus.cc:504] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } }
I20260926 20:23:20.392756 14723 leader_election.cc:288] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e [CANDIDATE]: Term 1 election: Requested vote from peers a410b1638b794d4b9f5475aee9fed5e1 (127.0.116.193:37055), f9aa02cf617040d78c5f86ce7f144961 (127.0.116.195:39271)
I20260926 20:23:20.393266 14654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2472efdebbe6475eb3d0531c876e2a13" candidate_uuid: "1baa9bc83c5e423198f49cef556af54e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9aa02cf617040d78c5f86ce7f144961"
I20260926 20:23:20.393296 14385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2472efdebbe6475eb3d0531c876e2a13" candidate_uuid: "1baa9bc83c5e423198f49cef556af54e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a410b1638b794d4b9f5475aee9fed5e1"
I20260926 20:23:20.393384 14654 raft_consensus.cc:3037] T 2472efdebbe6475eb3d0531c876e2a13 P f9aa02cf617040d78c5f86ce7f144961 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:23:20.393438 14385 raft_consensus.cc:3037] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:23:20.394551 14385 raft_consensus.cc:2454] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1baa9bc83c5e423198f49cef556af54e in term 1.
I20260926 20:23:20.394563 14654 raft_consensus.cc:2454] T 2472efdebbe6475eb3d0531c876e2a13 P f9aa02cf617040d78c5f86ce7f144961 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1baa9bc83c5e423198f49cef556af54e in term 1.
I20260926 20:23:20.394893 14454 leader_election.cc:302] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1baa9bc83c5e423198f49cef556af54e, f9aa02cf617040d78c5f86ce7f144961; no voters:
I20260926 20:23:20.395074 14723 raft_consensus.cc:2781] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:23:20.395351 14723 raft_consensus.cc:686] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e [term 1 LEADER]: Becoming Leader. State: Replica: 1baa9bc83c5e423198f49cef556af54e, State: Running, Role: LEADER
I20260926 20:23:20.395756 14723 consensus_queue.cc:237] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } }
I20260926 20:23:20.399138 14239 catalog_manager.cc:5495] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1baa9bc83c5e423198f49cef556af54e (127.0.116.194). New cstate: current_term: 1 leader_uuid: "1baa9bc83c5e423198f49cef556af54e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } health_report { overall_health: HEALTHY } } }
I20260926 20:23:20.426139 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:20.422376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43376 (local address 127.0.116.194:37991)
0926 20:23:20.422500 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:20.422504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:20.422517 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:23:20.422651 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:20.422653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:20.422730 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:20.422832 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:23:20.423139 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.423761 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.424656 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.424878 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.424981 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:20.425222 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:23:20.425743 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:23:20.425853 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:20.425934 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 20:23:20.425987 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 20:23:20.426396 14701 tablet.cc:2369] T 2472efdebbe6475eb3d0531c876e2a13 P f9aa02cf617040d78c5f86ce7f144961: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:23:20.432395 14385 raft_consensus.cc:1261] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Refusing update from remote peer 1baa9bc83c5e423198f49cef556af54e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:23:20.432404 14654 raft_consensus.cc:1261] T 2472efdebbe6475eb3d0531c876e2a13 P f9aa02cf617040d78c5f86ce7f144961 [term 1 FOLLOWER]: Refusing update from remote peer 1baa9bc83c5e423198f49cef556af54e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:23:20.433157 14727 consensus_queue.cc:1035] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:23:20.433338 14723 consensus_queue.cc:1035] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:23:20.441104 14731 mvcc.cc:205] Tried to move back new op lower bound from 7333700404963692544 to 7333700404824059904. Current Snapshot: MvccSnapshot[applied={T|T < 7333700404963692544}]
I20260926 20:23:20.456409 14239 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:23:20.456586 14239 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:23:20.459203 14239 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7: Sending DeleteTablet for 3 replicas of tablet 2472efdebbe6475eb3d0531c876e2a13
I20260926 20:23:20.460124 14500 tablet_service.cc:1514] Processing DeleteTablet for tablet 2472efdebbe6475eb3d0531c876e2a13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:23:20 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:43366
I20260926 20:23:20.460191 14634 tablet_service.cc:1514] Processing DeleteTablet for tablet 2472efdebbe6475eb3d0531c876e2a13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:23:20 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:34694
I20260926 20:23:20.460659 14741 tablet_replica.cc:330] T 2472efdebbe6475eb3d0531c876e2a13 P f9aa02cf617040d78c5f86ce7f144961: stopping tablet replica
I20260926 20:23:20.460589 14365 tablet_service.cc:1514] Processing DeleteTablet for tablet 2472efdebbe6475eb3d0531c876e2a13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:23:20 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:38220
I20260926 20:23:20.460966 14741 raft_consensus.cc:2229] T 2472efdebbe6475eb3d0531c876e2a13 P f9aa02cf617040d78c5f86ce7f144961 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:23:20.461114 14742 tablet_replica.cc:330] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1: stopping tablet replica
I20260926 20:23:20.461321 14741 raft_consensus.cc:2258] T 2472efdebbe6475eb3d0531c876e2a13 P f9aa02cf617040d78c5f86ce7f144961 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:23:20.461346 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 14302
I20260926 20:23:20.462267 14740 tablet_replica.cc:330] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e: stopping tablet replica
I20260926 20:23:20.462473 14740 raft_consensus.cc:2229] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:23:20.462770 14740 raft_consensus.cc:2258] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:23:20.464097 14740 ts_tablet_manager.cc:1905] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:23:20.466584 14740 ts_tablet_manager.cc:1918] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:23:20.466715 14740 log.cc:1196] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/2472efdebbe6475eb3d0531c876e2a13
I20260926 20:23:20.467060 14740 ts_tablet_manager.cc:1939] T 2472efdebbe6475eb3d0531c876e2a13 P 1baa9bc83c5e423198f49cef556af54e: Deleting consensus metadata
I20260926 20:23:20.468494 14741 ts_tablet_manager.cc:1905] T 2472efdebbe6475eb3d0531c876e2a13 P f9aa02cf617040d78c5f86ce7f144961: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:23:20.468506 14226 catalog_manager.cc:4841] TS 1baa9bc83c5e423198f49cef556af54e (127.0.116.194:37991): tablet 2472efdebbe6475eb3d0531c876e2a13 (table test-table [id=dd5c233419624ca9a1c210d616122c17]) successfully deleted
W20260926 20:23:20.469688 14225 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.116.193:37055 (error 108)
I20260926 20:23:20.470897 14711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:20.470573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:37055 (local address 127.0.0.1:38236)
0926 20:23:20.470726 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:20.470810 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.193:37055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 20:23:20.470920 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 14437
W20260926 20:23:20.472476 14225 catalog_manager.cc:4584] TS a410b1638b794d4b9f5475aee9fed5e1 (127.0.116.193:37055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2472efdebbe6475eb3d0531c876e2a13: Network error: Client connection negotiation failed: client connection to 127.0.116.193:37055: connect: Connection refused (error 111)
I20260926 20:23:20.472944 14741 ts_tablet_manager.cc:1918] T 2472efdebbe6475eb3d0531c876e2a13 P f9aa02cf617040d78c5f86ce7f144961: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:23:20.473063 14741 log.cc:1196] T 2472efdebbe6475eb3d0531c876e2a13 P f9aa02cf617040d78c5f86ce7f144961: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/2472efdebbe6475eb3d0531c876e2a13
I20260926 20:23:20.473366 14741 ts_tablet_manager.cc:1939] T 2472efdebbe6475eb3d0531c876e2a13 P f9aa02cf617040d78c5f86ce7f144961: Deleting consensus metadata
I20260926 20:23:20.474574 14225 catalog_manager.cc:4841] TS f9aa02cf617040d78c5f86ce7f144961 (127.0.116.195:39271): tablet 2472efdebbe6475eb3d0531c876e2a13 (table test-table [id=dd5c233419624ca9a1c210d616122c17]) successfully deleted
I20260926 20:23:20.476713 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 14571
I20260926 20:23:20.482210 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 14208
I20260926 20:23:20.489310 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:46601
--webserver_interface=127.0.116.254
--webserver_port=43999
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:46601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:23:20.606594 14744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:20.606932 14744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:20.607023 14744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:20.611253 14744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:23:20.611361 14744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:20.611398 14744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:20.611449 14744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:23:20.611479 14744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:23:20.617272 14744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:46601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:46601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=43999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:20.618453 14744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:20.619598 14744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:20.633704 14750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:20.633678 14753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:20.633742 14751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:20.634215 14744 server_base.cc:1034] running on GCE node
I20260926 20:23:20.634874 14744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:20.636260 14744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:20.637475 14744 hybrid_clock.cc:648] HybridClock initialized: now 1790454200637435 us; error 69 us; skew 500 ppm
Sep 26 20:23:20 dist-test-slave-qsm9 krb5kdc[14195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:20.640838 14744 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:23:20.641849 14744 webserver.cc:460] Webserver started at http://127.0.116.254:43999/ using document root <none> and password file <none>
I20260926 20:23:20.642486 14744 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:20.642601 14744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:20.646781 14744 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:23:20.649415 14760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:20.650929 14744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:23:20.651109 14744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "ff0e2eb98cb245028523b566380d48d7"
format_stamp: "Formatted at 2026-09-26 20:23:19 on dist-test-slave-qsm9"
server_key: "2f2fdd4a32dadad210078bb539135280"
server_key_iv: "437e56db8781c7ae4c2fcba32a58b57e"
server_key_version: "encryptionkey@0"
I20260926 20:23:20.651837 14744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:20.665239 14744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:20.669067 14744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:20.669323 14744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:20.676791 14744 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:46601
I20260926 20:23:20.677157 14812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:46601 every 8 connection(s)
I20260926 20:23:20.677977 14744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:23:20.682549 14813 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:23:20.684315 14813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7: Bootstrap starting.
I20260926 20:23:20.686447 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 14744
I20260926 20:23:20.686935 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:37055
--local_ip_for_outbound_sockets=127.0.116.193
--tserver_master_addrs=127.0.116.254:46601
--webserver_port=34077
--webserver_interface=127.0.116.193
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:23:20.702126 14813 log.cc:824] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7: Log is configured to *not* fsync() on all Append() calls
I20260926 20:23:20.711067 14813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:23:20.711476 14813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7: Bootstrap complete.
I20260926 20:23:20.715708 14813 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0e2eb98cb245028523b566380d48d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46601 } }
I20260926 20:23:20.716140 14813 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff0e2eb98cb245028523b566380d48d7, State: Initialized, Role: FOLLOWER
I20260926 20:23:20.716686 14813 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0e2eb98cb245028523b566380d48d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46601 } }
I20260926 20:23:20.716807 14813 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:23:20.716897 14813 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:23:20.717010 14813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:23:20.719129 14813 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0e2eb98cb245028523b566380d48d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46601 } }
I20260926 20:23:20.719558 14813 leader_election.cc:302] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff0e2eb98cb245028523b566380d48d7; no voters:
I20260926 20:23:20.719931 14813 leader_election.cc:288] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:23:20.720041 14817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:23:20.720254 14817 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 2 LEADER]: Becoming Leader. State: Replica: ff0e2eb98cb245028523b566380d48d7, State: Running, Role: LEADER
I20260926 20:23:20.720580 14817 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0e2eb98cb245028523b566380d48d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46601 } }
I20260926 20:23:20.721294 14813 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:23:20.722543 14818 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ff0e2eb98cb245028523b566380d48d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0e2eb98cb245028523b566380d48d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46601 } } }
I20260926 20:23:20.722695 14818 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [sys.catalog]: This master's current role is: LEADER
I20260926 20:23:20.723094 14819 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff0e2eb98cb245028523b566380d48d7. Latest consensus state: current_term: 2 leader_uuid: "ff0e2eb98cb245028523b566380d48d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0e2eb98cb245028523b566380d48d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46601 } } }
I20260926 20:23:20.723183 14819 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [sys.catalog]: This master's current role is: LEADER
I20260926 20:23:20.727308 14833 catalog_manager.cc:1259] Loaded cluster ID: 977edc0f6d874980b3c25b6b277af0fd
I20260926 20:23:20.727396 14833 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7: loading cluster ID for follower catalog manager: success
W20260926 20:23:20.728437 14833 catalog_manager.cc:1576] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260926 20:23:20.728525 14833 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260926 20:23:20.728585 14824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:23:20.731101 14824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:23:20.731660 14824 catalog_manager.cc:1259] Loaded cluster ID: 977edc0f6d874980b3c25b6b277af0fd
I20260926 20:23:20.731726 14824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:23:20.732834 14824 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 20:23:20.737754 14824 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 20:23:20.743829 14824 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:23:20.744697 14824 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7: Loaded TSK: 0
I20260926 20:23:20.745965 14824 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:23:20.807770 14815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:20.808115 14815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:20.808202 14815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:20.812662 14815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:20.812799 14815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:20.812918 14815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:23:20.818797 14815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:37055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=34077
--tserver_master_addrs=127.0.116.254:46601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:20.820035 14815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:20.821141 14815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:20.831871 14841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:20.831871 14840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:20.831898 14843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:20.832973 14815 server_base.cc:1034] running on GCE node
I20260926 20:23:20.833482 14815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:20.834198 14815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:20.835388 14815 hybrid_clock.cc:648] HybridClock initialized: now 1790454200835371 us; error 37 us; skew 500 ppm
Sep 26 20:23:20 dist-test-slave-qsm9 krb5kdc[14195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:20.838577 14815 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:23:20.839745 14815 webserver.cc:460] Webserver started at http://127.0.116.193:34077/ using document root <none> and password file <none>
I20260926 20:23:20.840344 14815 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:20.840456 14815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:20.844681 14815 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:23:20.846999 14850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:20.848479 14815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:23:20.848613 14815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "a410b1638b794d4b9f5475aee9fed5e1"
format_stamp: "Formatted at 2026-09-26 20:23:19 on dist-test-slave-qsm9"
server_key: "c3d457adf56dacf4d7656ebcb94a0e42"
server_key_iv: "d2f83fbdc0d1eb4b3d4a8299d0e99ebc"
server_key_version: "encryptionkey@0"
I20260926 20:23:20.849206 14815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:20.867775 14815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:20.871095 14815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:20.871306 14815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:20.872064 14815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:23:20.873227 14857 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:23:20.877547 14815 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:23:20.877620 14815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260926 20:23:20.877672 14815 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:23:20.879413 14815 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:23:20.879479 14815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 20:23:20.879773 14857 tablet_bootstrap.cc:492] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1: Bootstrap starting.
I20260926 20:23:20.890723 14815 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:37055
I20260926 20:23:20.890774 14964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:37055 every 8 connection(s)
I20260926 20:23:20.892045 14815 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:23:20.894364 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 14815
I20260926 20:23:20.894867 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:37991
--local_ip_for_outbound_sockets=127.0.116.194
--tserver_master_addrs=127.0.116.254:46601
--webserver_port=33111
--webserver_interface=127.0.116.194
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:23:20.901957 14857 log.cc:824] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:23:20 dist-test-slave-qsm9 krb5kdc[14195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:23:20.910889 14857 tablet_bootstrap.cc:492] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:23:20.911352 14857 tablet_bootstrap.cc:492] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1: Bootstrap complete.
I20260926 20:23:20.912292 14968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:20.894304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.193:36531)
0926 20:23:20.895368 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:20.895430 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:20.896641 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:20.897806 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:20.897814 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:20.898251 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:20.898732 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:20.898746 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.900268 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.900271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:20.901352 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:20.901383 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.902397 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.902400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:20.902615 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:20.903446 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:20.903468 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:20.905602 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:20.908564 (+ 2962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:20.908574 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:20.908591 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:20.909000 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:20.909417 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:20.909422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:20.909424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:20.909597 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:20.910331 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:20.910337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:20.910639 (+ 302us) client_negotiation.cc:769] Sending connection context
0926 20:23:20.910837 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 20:23:20.911181 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":133,"threads_started":1}
I20260926 20:23:20.912783 14857 ts_tablet_manager.cc:1397] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.007s
I20260926 20:23:20.912986 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:20.894336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:36531 (local address 127.0.116.254:46601)
0926 20:23:20.894774 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:20.894799 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:20.895580 (+ 781us) server_negotiation.cc:407] Connection header received
0926 20:23:20.897076 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:20.897087 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:20.897401 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:20.897743 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:20.898913 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.900057 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.901531 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.902027 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.902185 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:20.905851 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:20.905857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:20.905876 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:20.905913 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:20.908415 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:20.909134 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:20.909141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:20.909153 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:20.909314 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:20.909697 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:20.909700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:20.909702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:20.910169 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:20.910305 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:20.911113 (+ 808us) server_negotiation.cc:299] Negotiation successful
0926 20:23:20.911640 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":112,"threads_started":1}
I20260926 20:23:20.915481 14965 heartbeater.cc:347] Connected to a master server at 127.0.116.254:46601
I20260926 20:23:20.915753 14965 heartbeater.cc:464] Registering TS with master...
I20260926 20:23:20.915555 14857 raft_consensus.cc:348] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } }
I20260926 20:23:20.916075 14857 raft_consensus.cc:729] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a410b1638b794d4b9f5475aee9fed5e1, State: Initialized, Role: FOLLOWER
I20260926 20:23:20.916332 14965 heartbeater.cc:511] Master 127.0.116.254:46601 requested a full tablet report, sending...
I20260926 20:23:20.916554 14857 consensus_queue.cc:260] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } }
I20260926 20:23:20.917695 14857 ts_tablet_manager.cc:1428] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 20:23:20.919375 14777 ts_manager.cc:194] Registered new tserver with Master: a410b1638b794d4b9f5475aee9fed5e1 (127.0.116.193:37055)
I20260926 20:23:20.924070 14777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:36531
I20260926 20:23:20.926759 14965 heartbeater.cc:503] Master 127.0.116.254:46601 was elected leader, sending a full tablet report...
Sep 26 20:23:20 dist-test-slave-qsm9 krb5kdc[14195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for kudu/127.0.116.193@KRBTEST.COM
I20260926 20:23:20.935456 14972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:20.922962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:37055 (local address 127.0.0.1:38246)
0926 20:23:20.923345 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:20.923388 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:20.923587 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:20.925449 (+ 1862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:20.925455 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:20.925956 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:20.926082 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:20.926090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.927783 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.927787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:20.928817 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:20.928830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.929548 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.929550 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:20.929626 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:20.930187 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:20.930198 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:20.931740 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:20.933796 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:20.933806 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:20.933812 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:20.934031 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:20.934385 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:20.934387 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:20.934389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:20.934463 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:20.934931 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:20.934935 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:20.935034 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 20:23:20.935141 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 20:23:20.935319 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":128,"threads_started":1}
I20260926 20:23:20.935835 14973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:20.923052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.0.116.193:37055)
0926 20:23:20.924971 (+ 1919us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:20.924985 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:20.925009 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:23:20.925077 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:20.925082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:20.925288 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:20.925425 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:20.926582 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.927673 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.928935 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:20.929441 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:20.929552 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:20.931878 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:20.931887 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:20.931901 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:20.931929 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:20.933690 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:20.934135 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:20.934141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:20.934143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:20.934270 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:20.934542 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:20.934545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:20.934547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:20.934832 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:20.934962 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:20.935251 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 20:23:20.935407 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1821,"thread_start_us":112,"threads_started":1}
I20260926 20:23:20.936841 14899 tablet_service.cc:1514] Processing DeleteTablet for tablet 2472efdebbe6475eb3d0531c876e2a13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:23:20 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:38246
I20260926 20:23:20.937682 14974 tablet_replica.cc:330] stopping tablet replica
I20260926 20:23:20.937937 14974 raft_consensus.cc:2229] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:23:20.938138 14974 raft_consensus.cc:2258] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:23:20.939114 14974 ts_tablet_manager.cc:1905] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:23:20.942730 14974 ts_tablet_manager.cc:1918] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:23:20.942823 14974 log.cc:1196] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/wal/wals/2472efdebbe6475eb3d0531c876e2a13
I20260926 20:23:20.943102 14974 ts_tablet_manager.cc:1939] T 2472efdebbe6475eb3d0531c876e2a13 P a410b1638b794d4b9f5475aee9fed5e1: Deleting consensus metadata
I20260926 20:23:20.944945 14762 catalog_manager.cc:4841] TS a410b1638b794d4b9f5475aee9fed5e1 (127.0.116.193:37055): tablet 2472efdebbe6475eb3d0531c876e2a13 (table test-table [id=dd5c233419624ca9a1c210d616122c17]) successfully deleted
W20260926 20:23:21.021648 14969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:21.022040 14969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:21.022094 14969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:21.026365 14969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:21.026438 14969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:21.026500 14969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:23:21.032102 14969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:37991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=33111
--tserver_master_addrs=127.0.116.254:46601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:21.033190 14969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:21.034291 14969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:21.044638 14980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:21.044761 14969 server_base.cc:1034] running on GCE node
W20260926 20:23:21.044623 14979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:21.044623 14982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:21.045375 14969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:21.046025 14969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:21.047197 14969 hybrid_clock.cc:648] HybridClock initialized: now 1790454201047181 us; error 30 us; skew 500 ppm
Sep 26 20:23:21 dist-test-slave-qsm9 krb5kdc[14195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:21.050282 14969 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:23:21.051242 14969 webserver.cc:460] Webserver started at http://127.0.116.194:33111/ using document root <none> and password file <none>
I20260926 20:23:21.051867 14969 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:21.051975 14969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:21.056190 14969 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:23:21.058423 14989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:21.059828 14969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 20:23:21.059957 14969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "1baa9bc83c5e423198f49cef556af54e"
format_stamp: "Formatted at 2026-09-26 20:23:19 on dist-test-slave-qsm9"
server_key: "40d8ec69c428090b1d549185bc50694b"
server_key_iv: "f484f6529cc4173b2088ba8c5c45026b"
server_key_version: "encryptionkey@0"
I20260926 20:23:21.060523 14969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:21.094974 14969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:21.098495 14969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:21.098703 14969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:21.099313 14969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:23:21.100282 14969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:23:21.100342 14969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:21.100407 14969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:23:21.100437 14969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:21.109793 14969 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:37991
I20260926 20:23:21.109812 15102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:37991 every 8 connection(s)
I20260926 20:23:21.110850 14969 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:23:21.115286 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 14969
I20260926 20:23:21.115797 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:39271
--local_ip_for_outbound_sockets=127.0.116.195
--tserver_master_addrs=127.0.116.254:46601
--webserver_port=40359
--webserver_interface=127.0.116.195
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:23:21 dist-test-slave-qsm9 krb5kdc[14195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:23:21.126605 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:21.112812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:55633 (local address 127.0.116.254:46601)
0926 20:23:21.112987 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:21.112992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:21.114154 (+ 1162us) server_negotiation.cc:407] Connection header received
0926 20:23:21.115067 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:21.115070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:21.115137 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:21.115266 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:21.116722 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.117424 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.118970 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.119211 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.119327 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:21.122199 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:21.122205 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:21.122211 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:21.122253 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:21.124029 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:21.124738 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:21.124741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:21.124743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:21.124827 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:21.125246 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:21.125248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:21.125250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:21.125451 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:21.125525 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:21.126329 (+ 804us) server_negotiation.cc:299] Negotiation successful
0926 20:23:21.126476 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:23:21.127873 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:21.113265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.194:55633)
0926 20:23:21.114001 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:21.114044 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:21.114852 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:21.115457 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:21.115466 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:21.116051 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:21.116557 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:21.116571 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.117567 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.117573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:21.118841 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:21.118851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.119373 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.119377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:21.119604 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:21.120249 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:21.120264 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:21.122010 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:21.124154 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:21.124163 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:21.124185 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:21.124615 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:21.124923 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:21.124928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:21.124933 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:21.125139 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:21.125556 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:21.125564 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:21.125964 (+ 400us) client_negotiation.cc:769] Sending connection context
0926 20:23:21.126230 (+ 266us) client_negotiation.cc:240] Negotiation successful
0926 20:23:21.126652 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":107,"threads_started":1}
I20260926 20:23:21.129103 15103 heartbeater.cc:347] Connected to a master server at 127.0.116.254:46601
I20260926 20:23:21.129372 15103 heartbeater.cc:464] Registering TS with master...
I20260926 20:23:21.129904 15103 heartbeater.cc:511] Master 127.0.116.254:46601 requested a full tablet report, sending...
I20260926 20:23:21.130977 14777 ts_manager.cc:194] Registered new tserver with Master: 1baa9bc83c5e423198f49cef556af54e (127.0.116.194:37991)
I20260926 20:23:21.131635 14777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:55633
W20260926 20:23:21.232533 15106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:21.232837 15106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:21.232916 15106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:21.237037 15106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:21.237128 15106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:21.237219 15106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:23:21.243042 15106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:39271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=40359
--tserver_master_addrs=127.0.116.254:46601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:21.244308 15106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:21.245329 15106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:21.255074 15114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:21.255096 15112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:21.255121 15111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:21.255434 15106 server_base.cc:1034] running on GCE node
I20260926 20:23:21.255898 15106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:21.256556 15106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:21.257754 15106 hybrid_clock.cc:648] HybridClock initialized: now 1790454201257726 us; error 43 us; skew 500 ppm
Sep 26 20:23:21 dist-test-slave-qsm9 krb5kdc[14195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:21.260598 15106 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:23:21.261493 15106 webserver.cc:460] Webserver started at http://127.0.116.195:40359/ using document root <none> and password file <none>
I20260926 20:23:21.262087 15106 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:21.262171 15106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:21.266136 15106 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:23:21.268311 15121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:21.269671 15106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:23:21.269810 15106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "f9aa02cf617040d78c5f86ce7f144961"
format_stamp: "Formatted at 2026-09-26 20:23:20 on dist-test-slave-qsm9"
server_key: "35dd14c426d0249f64d795ba41e0d4ea"
server_key_iv: "8a2e3bee538f3b0d9aa0772aa62177d2"
server_key_version: "encryptionkey@0"
I20260926 20:23:21.270330 15106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:21.282950 15106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:21.286234 15106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:21.286409 15106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:21.286969 15106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:23:21.287834 15106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:23:21.287882 15106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:21.287922 15106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:23:21.287936 15106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:21.297375 15106 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:39271
I20260926 20:23:21.297397 15234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:39271 every 8 connection(s)
I20260926 20:23:21.298492 15106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:23:21.302802 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 15106
Sep 26 20:23:21 dist-test-slave-qsm9 krb5kdc[14195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:23:21.312927 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:21.303596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42178 (local address 127.0.116.254:46601)
0926 20:23:21.303860 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:21.303865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:21.304037 (+ 172us) server_negotiation.cc:407] Connection header received
0926 20:23:21.304197 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:21.304204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:21.304246 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:21.304340 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:21.305232 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.306053 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.306983 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.307309 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.307431 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:21.308841 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:21.308847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:21.308851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:21.308882 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:21.310998 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:21.311554 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:21.311561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:21.311564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:21.311655 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:21.311930 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:21.311934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:21.311936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:21.312181 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:21.312346 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:21.312565 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 20:23:21.312699 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":184,"thread_start_us":98,"threads_started":1}
I20260926 20:23:21.314395 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:21.300558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:46795 (local address 127.0.116.254:46601)
0926 20:23:21.300718 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:21.300722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:21.301868 (+ 1146us) server_negotiation.cc:407] Connection header received
0926 20:23:21.302827 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:21.302833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:21.302897 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:21.302997 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:21.304552 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.305353 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.306795 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.307103 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.307179 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:21.310265 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:21.310270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:21.310276 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:21.310322 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:21.312062 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:21.312690 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:21.312694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:21.312695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:21.312767 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:21.313179 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:21.313183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:21.313184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:21.313413 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:21.313503 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:21.314130 (+ 627us) server_negotiation.cc:299] Negotiation successful
0926 20:23:21.314243 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":56}
I20260926 20:23:21.315429 15237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:21.301005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.195:46795)
0926 20:23:21.301709 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:21.301755 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:21.302557 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:21.303280 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:21.303292 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:21.303861 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:21.304408 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:21.304423 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.305479 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.305483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:21.306660 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:21.306671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.307276 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.307281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:21.307542 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:21.308251 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:21.308268 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:21.310088 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:21.312165 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:21.312170 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:21.312185 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:21.312598 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:21.312884 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:21.312887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:21.312889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:21.313081 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:21.313507 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:21.313516 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:21.313874 (+ 358us) client_negotiation.cc:769] Sending connection context
0926 20:23:21.314093 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 20:23:21.314459 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":126,"threads_started":1}
I20260926 20:23:21.316574 15235 heartbeater.cc:347] Connected to a master server at 127.0.116.254:46601
I20260926 20:23:21.316818 15235 heartbeater.cc:464] Registering TS with master...
I20260926 20:23:21.317313 15235 heartbeater.cc:511] Master 127.0.116.254:46601 requested a full tablet report, sending...
I20260926 20:23:21.318369 14777 ts_manager.cc:194] Registered new tserver with Master: f9aa02cf617040d78c5f86ce7f144961 (127.0.116.195:39271)
I20260926 20:23:21.318965 14777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:46795
I20260926 20:23:21.326952 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:23:21.331627 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:21.327705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50836 (local address 127.0.116.254:46601)
0926 20:23:21.327827 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:21.327831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:21.328082 (+ 251us) server_negotiation.cc:407] Connection header received
0926 20:23:21.328230 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:21.328232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:21.328367 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:21.328459 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:23:21.328673 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.329191 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.330065 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.330265 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.330338 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:21.330667 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:23:21.331104 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:23:21.331225 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:21.331362 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 20:23:21.331430 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:23:21.332294 14777 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:50836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:23:21.335764 14777 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:23:21.344578 14899 tablet_service.cc:1467] Processing CreateTablet for tablet e515f04999f249d3b628f4f2ea14a8af (DEFAULT_TABLE table=test-table [id=e61385e9d1714635aca443d4380851bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:23:21.345350 14899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e515f04999f249d3b628f4f2ea14a8af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:23:21.348466 14857 tablet_bootstrap.cc:492] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1: Bootstrap starting.
I20260926 20:23:21.349032 14972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:21.343047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:37991 (local address 127.0.0.1:54992)
0926 20:23:21.343173 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:21.343190 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:21.343270 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:21.344190 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:21.344193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:21.344211 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:23:21.344334 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:21.344340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.346047 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.346053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:21.347229 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:21.347241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.348536 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.348539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:21.348646 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:21.348697 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 20:23:21.348775 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:23:21.348863 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 20:23:21.349114 15244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:21.343224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.0.116.194:37991)
0926 20:23:21.343588 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:21.343603 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:21.343628 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:23:21.343712 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:21.343718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:21.343976 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:21.344200 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:23:21.344436 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.345906 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.347414 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.348333 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.348488 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:21.348665 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:21.348862 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 20:23:21.348973 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":152,"threads_started":1}
I20260926 20:23:21.349386 14857 tablet_bootstrap.cc:654] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1: Neither blocks nor log segments found. Creating new log.
I20260926 20:23:21.349397 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:21.343692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52994 (local address 127.0.116.195:39271)
0926 20:23:21.344125 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:21.344142 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:21.344164 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:23:21.344237 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:21.344242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:21.344447 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:21.344620 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:23:21.344814 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.346136 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.347290 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.348292 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.348551 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:21.348701 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:21.349103 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 20:23:21.349217 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":134,"threads_started":1}
I20260926 20:23:21.349615 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:21.343525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:39271 (local address 127.0.0.1:52994)
0926 20:23:21.343764 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:21.343776 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:21.343875 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:21.344603 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:21.344608 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:21.344630 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:23:21.344718 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:21.344724 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.346271 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.346274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:21.347158 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:21.347168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.348769 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.348773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:21.348854 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:21.348885 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 20:23:21.348952 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 20:23:21.349016 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":1,"thread_start_us":119,"threads_started":1}
I20260926 20:23:21.351009 14857 tablet_bootstrap.cc:492] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1: No bootstrap required, opened a new log
I20260926 20:23:21.351096 14857 ts_tablet_manager.cc:1397] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:23:21.351549 14857 raft_consensus.cc:348] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } }
I20260926 20:23:21.351665 14857 raft_consensus.cc:374] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:23:21.351701 14857 raft_consensus.cc:729] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a410b1638b794d4b9f5475aee9fed5e1, State: Initialized, Role: FOLLOWER
I20260926 20:23:21.351841 14857 consensus_queue.cc:260] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } }
I20260926 20:23:21.352782 14857 ts_tablet_manager.cc:1428] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:23:21.352739 15169 tablet_service.cc:1467] Processing CreateTablet for tablet e515f04999f249d3b628f4f2ea14a8af (DEFAULT_TABLE table=test-table [id=e61385e9d1714635aca443d4380851bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:23:21.352692 15037 tablet_service.cc:1467] Processing CreateTablet for tablet e515f04999f249d3b628f4f2ea14a8af (DEFAULT_TABLE table=test-table [id=e61385e9d1714635aca443d4380851bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:23:21.353765 15037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e515f04999f249d3b628f4f2ea14a8af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:23:21.353796 15169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e515f04999f249d3b628f4f2ea14a8af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:23:21.361738 15248 tablet_bootstrap.cc:492] T e515f04999f249d3b628f4f2ea14a8af P 1baa9bc83c5e423198f49cef556af54e: Bootstrap starting.
I20260926 20:23:21.361732 15249 tablet_bootstrap.cc:492] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961: Bootstrap starting.
I20260926 20:23:21.364396 15248 tablet_bootstrap.cc:654] T e515f04999f249d3b628f4f2ea14a8af P 1baa9bc83c5e423198f49cef556af54e: Neither blocks nor log segments found. Creating new log.
I20260926 20:23:21.364396 15249 tablet_bootstrap.cc:654] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961: Neither blocks nor log segments found. Creating new log.
I20260926 20:23:21.365252 15249 log.cc:824] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961: Log is configured to *not* fsync() on all Append() calls
I20260926 20:23:21.365293 15248 log.cc:824] T e515f04999f249d3b628f4f2ea14a8af P 1baa9bc83c5e423198f49cef556af54e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:23:21.367679 15248 tablet_bootstrap.cc:492] T e515f04999f249d3b628f4f2ea14a8af P 1baa9bc83c5e423198f49cef556af54e: No bootstrap required, opened a new log
I20260926 20:23:21.367679 15249 tablet_bootstrap.cc:492] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961: No bootstrap required, opened a new log
I20260926 20:23:21.367908 15248 ts_tablet_manager.cc:1397] T e515f04999f249d3b628f4f2ea14a8af P 1baa9bc83c5e423198f49cef556af54e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:23:21.367908 15249 ts_tablet_manager.cc:1397] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:23:21.372124 15249 raft_consensus.cc:348] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } }
I20260926 20:23:21.372124 15248 raft_consensus.cc:348] T e515f04999f249d3b628f4f2ea14a8af P 1baa9bc83c5e423198f49cef556af54e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } }
I20260926 20:23:21.372457 15249 raft_consensus.cc:374] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:23:21.372457 15248 raft_consensus.cc:374] T e515f04999f249d3b628f4f2ea14a8af P 1baa9bc83c5e423198f49cef556af54e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:23:21.372519 15248 raft_consensus.cc:729] T e515f04999f249d3b628f4f2ea14a8af P 1baa9bc83c5e423198f49cef556af54e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1baa9bc83c5e423198f49cef556af54e, State: Initialized, Role: FOLLOWER
I20260926 20:23:21.372519 15249 raft_consensus.cc:729] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9aa02cf617040d78c5f86ce7f144961, State: Initialized, Role: FOLLOWER
I20260926 20:23:21.373036 15249 consensus_queue.cc:260] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } }
I20260926 20:23:21.373037 15248 consensus_queue.cc:260] T e515f04999f249d3b628f4f2ea14a8af P 1baa9bc83c5e423198f49cef556af54e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } }
I20260926 20:23:21.373772 15103 heartbeater.cc:503] Master 127.0.116.254:46601 was elected leader, sending a full tablet report...
I20260926 20:23:21.374128 15235 heartbeater.cc:503] Master 127.0.116.254:46601 was elected leader, sending a full tablet report...
I20260926 20:23:21.374171 15248 ts_tablet_manager.cc:1428] T e515f04999f249d3b628f4f2ea14a8af P 1baa9bc83c5e423198f49cef556af54e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:23:21.374171 15249 ts_tablet_manager.cc:1428] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260926 20:23:21.393373 14966 tablet.cc:2369] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:23:21.410471 15252 raft_consensus.cc:482] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:23:21.410737 15252 raft_consensus.cc:504] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } }
I20260926 20:23:21.412165 15252 leader_election.cc:288] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1baa9bc83c5e423198f49cef556af54e (127.0.116.194:37991), a410b1638b794d4b9f5475aee9fed5e1 (127.0.116.193:37055)
I20260926 20:23:21.417631 15244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:21.412500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:42517 (local address 127.0.116.194:37991)
0926 20:23:21.412631 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:21.412635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:21.412914 (+ 279us) server_negotiation.cc:407] Connection header received
0926 20:23:21.413013 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:21.413016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:21.413072 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:21.413134 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:23:21.413579 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.414374 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.416286 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.416916 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.417064 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:21.417165 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:21.417419 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 20:23:21.417497 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:23:21.417686 14973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:21.412512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:35153 (local address 127.0.116.193:37055)
0926 20:23:21.412692 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:21.412698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:21.412715 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:23:21.412778 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:21.412780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:21.412834 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:21.412949 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:23:21.413581 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.414368 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.416264 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.417006 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.417112 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:21.417260 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:21.417419 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 20:23:21.417497 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:23:21.417677 15237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:21.412378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:37055 (local address 127.0.116.195:35153)
0926 20:23:21.412578 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:21.412594 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:21.412695 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:21.413146 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:21.413149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:21.413190 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:23:21.413463 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:21.413470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.414488 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.414491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:21.416117 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:21.416129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.417109 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.417113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:21.417238 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:21.417302 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 20:23:21.417378 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 20:23:21.417510 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":307,"spinlock_wait_cycles":17024}
I20260926 20:23:21.417685 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:21.412379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:37991 (local address 127.0.116.195:42517)
0926 20:23:21.412821 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:21.412832 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:21.412928 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:21.413153 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:21.413155 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:21.413191 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:23:21.413464 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:21.413470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.414488 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.414491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:21.416117 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:21.416129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.417077 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.417083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:21.417233 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:21.417302 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 20:23:21.417378 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 20:23:21.417510 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":352,"spinlock_wait_cycles":36096,"thread_start_us":139,"threads_started":1}
I20260926 20:23:21.418269 14919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e515f04999f249d3b628f4f2ea14a8af" candidate_uuid: "f9aa02cf617040d78c5f86ce7f144961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a410b1638b794d4b9f5475aee9fed5e1" is_pre_election: true
I20260926 20:23:21.418367 15057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e515f04999f249d3b628f4f2ea14a8af" candidate_uuid: "f9aa02cf617040d78c5f86ce7f144961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1baa9bc83c5e423198f49cef556af54e" is_pre_election: true
I20260926 20:23:21.418588 14919 raft_consensus.cc:2454] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9aa02cf617040d78c5f86ce7f144961 in term 0.
I20260926 20:23:21.418697 15057 raft_consensus.cc:2454] T e515f04999f249d3b628f4f2ea14a8af P 1baa9bc83c5e423198f49cef556af54e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9aa02cf617040d78c5f86ce7f144961 in term 0.
I20260926 20:23:21.419117 15123 leader_election.cc:302] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1baa9bc83c5e423198f49cef556af54e, a410b1638b794d4b9f5475aee9fed5e1, f9aa02cf617040d78c5f86ce7f144961; no voters:
I20260926 20:23:21.419394 15252 raft_consensus.cc:2781] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:23:21.419483 15252 raft_consensus.cc:482] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:23:21.419554 15252 raft_consensus.cc:3037] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:23:21.420578 15252 raft_consensus.cc:504] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } }
I20260926 20:23:21.420909 15252 leader_election.cc:288] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961 [CANDIDATE]: Term 1 election: Requested vote from peers 1baa9bc83c5e423198f49cef556af54e (127.0.116.194:37991), a410b1638b794d4b9f5475aee9fed5e1 (127.0.116.193:37055)
I20260926 20:23:21.421216 15057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e515f04999f249d3b628f4f2ea14a8af" candidate_uuid: "f9aa02cf617040d78c5f86ce7f144961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1baa9bc83c5e423198f49cef556af54e"
I20260926 20:23:21.421250 14919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e515f04999f249d3b628f4f2ea14a8af" candidate_uuid: "f9aa02cf617040d78c5f86ce7f144961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a410b1638b794d4b9f5475aee9fed5e1"
I20260926 20:23:21.421360 15057 raft_consensus.cc:3037] T e515f04999f249d3b628f4f2ea14a8af P 1baa9bc83c5e423198f49cef556af54e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:23:21.421398 14919 raft_consensus.cc:3037] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:23:21.422544 14919 raft_consensus.cc:2454] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9aa02cf617040d78c5f86ce7f144961 in term 1.
I20260926 20:23:21.422758 15057 raft_consensus.cc:2454] T e515f04999f249d3b628f4f2ea14a8af P 1baa9bc83c5e423198f49cef556af54e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9aa02cf617040d78c5f86ce7f144961 in term 1.
I20260926 20:23:21.422838 15123 leader_election.cc:302] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a410b1638b794d4b9f5475aee9fed5e1, f9aa02cf617040d78c5f86ce7f144961; no voters:
I20260926 20:23:21.423009 15252 raft_consensus.cc:2781] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:23:21.423224 15252 raft_consensus.cc:686] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961 [term 1 LEADER]: Becoming Leader. State: Replica: f9aa02cf617040d78c5f86ce7f144961, State: Running, Role: LEADER
I20260926 20:23:21.423609 15252 consensus_queue.cc:237] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } }
I20260926 20:23:21.427121 14776 catalog_manager.cc:5495] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9aa02cf617040d78c5f86ce7f144961 (127.0.116.195). New cstate: current_term: 1 leader_uuid: "f9aa02cf617040d78c5f86ce7f144961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } health_report { overall_health: UNKNOWN } } }
I20260926 20:23:21.455627 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:21.451984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52996 (local address 127.0.116.195:39271)
0926 20:23:21.452113 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:21.452117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:21.452131 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:23:21.452301 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:21.452303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:21.452386 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:21.452482 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:23:21.452678 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.453242 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.454077 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:21.454290 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:21.454395 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:21.454702 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:23:21.455158 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:23:21.455249 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:21.455375 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 20:23:21.455444 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:23:21.461867 14919 raft_consensus.cc:1261] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Refusing update from remote peer f9aa02cf617040d78c5f86ce7f144961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:23:21.461881 15057 raft_consensus.cc:1261] T e515f04999f249d3b628f4f2ea14a8af P 1baa9bc83c5e423198f49cef556af54e [term 1 FOLLOWER]: Refusing update from remote peer f9aa02cf617040d78c5f86ce7f144961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:23:21.462601 15255 consensus_queue.cc:1035] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:23:21.462818 15252 consensus_queue.cc:1035] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:23:21.469722 15258 mvcc.cc:205] Tried to move back new op lower bound from 7333700409180463104 to 7333700409034117120. Current Snapshot: MvccSnapshot[applied={T|T < 7333700409180463104}]
I20260926 20:23:21.469750 15259 mvcc.cc:205] Tried to move back new op lower bound from 7333700409180463104 to 7333700409034117120. Current Snapshot: MvccSnapshot[applied={T|T < 7333700409180463104}]
I20260926 20:23:21.471350 15260 mvcc.cc:205] Tried to move back new op lower bound from 7333700409180463104 to 7333700409034117120. Current Snapshot: MvccSnapshot[applied={T|T < 7333700409180463104}]
I20260926 20:23:21.479128 14775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:50836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:23:21.479316 14775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:50836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:23:21.481912 14775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7: Sending DeleteTablet for 3 replicas of tablet e515f04999f249d3b628f4f2ea14a8af
I20260926 20:23:21.482703 15169 tablet_service.cc:1514] Processing DeleteTablet for tablet e515f04999f249d3b628f4f2ea14a8af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:23:21 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:52994
I20260926 20:23:21.482678 15037 tablet_service.cc:1514] Processing DeleteTablet for tablet e515f04999f249d3b628f4f2ea14a8af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:23:21 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:54992
I20260926 20:23:21.483206 14899 tablet_service.cc:1514] Processing DeleteTablet for tablet e515f04999f249d3b628f4f2ea14a8af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:23:21 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:38246
I20260926 20:23:21.483249 15268 tablet_replica.cc:330] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961: stopping tablet replica
I20260926 20:23:21.483438 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 14815
I20260926 20:23:21.483594 15267 tablet_replica.cc:330] T e515f04999f249d3b628f4f2ea14a8af P 1baa9bc83c5e423198f49cef556af54e: stopping tablet replica
I20260926 20:23:21.484206 15267 raft_consensus.cc:2229] T e515f04999f249d3b628f4f2ea14a8af P 1baa9bc83c5e423198f49cef556af54e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:23:21.484355 15268 raft_consensus.cc:2229] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:23:21.484753 15267 raft_consensus.cc:2258] T e515f04999f249d3b628f4f2ea14a8af P 1baa9bc83c5e423198f49cef556af54e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:23:21.485211 15268 raft_consensus.cc:2258] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:23:21.487048 15268 ts_tablet_manager.cc:1905] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:23:21.487048 15267 ts_tablet_manager.cc:1905] T e515f04999f249d3b628f4f2ea14a8af P 1baa9bc83c5e423198f49cef556af54e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:23:21.489921 15267 ts_tablet_manager.cc:1918] T e515f04999f249d3b628f4f2ea14a8af P 1baa9bc83c5e423198f49cef556af54e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:23:21.489921 15268 ts_tablet_manager.cc:1918] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:23:21.490044 15267 log.cc:1196] T e515f04999f249d3b628f4f2ea14a8af P 1baa9bc83c5e423198f49cef556af54e: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/e515f04999f249d3b628f4f2ea14a8af
I20260926 20:23:21.490170 15268 log.cc:1196] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/e515f04999f249d3b628f4f2ea14a8af
I20260926 20:23:21.490424 15267 ts_tablet_manager.cc:1939] T e515f04999f249d3b628f4f2ea14a8af P 1baa9bc83c5e423198f49cef556af54e: Deleting consensus metadata
I20260926 20:23:21.490653 15268 ts_tablet_manager.cc:1939] T e515f04999f249d3b628f4f2ea14a8af P f9aa02cf617040d78c5f86ce7f144961: Deleting consensus metadata
W20260926 20:23:21.491394 14762 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.116.193:37055 (error 108)
I20260926 20:23:21.491943 14762 catalog_manager.cc:4841] TS f9aa02cf617040d78c5f86ce7f144961 (127.0.116.195:39271): tablet e515f04999f249d3b628f4f2ea14a8af (table test-table [id=e61385e9d1714635aca443d4380851bd]) successfully deleted
I20260926 20:23:21.491936 14763 catalog_manager.cc:4841] TS 1baa9bc83c5e423198f49cef556af54e (127.0.116.194:37991): tablet e515f04999f249d3b628f4f2ea14a8af (table test-table [id=e61385e9d1714635aca443d4380851bd]) successfully deleted
I20260926 20:23:21.492671 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 14969
I20260926 20:23:21.494055 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:21.493695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:37055 (local address 127.0.0.1:47902)
0926 20:23:21.493894 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:21.493962 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.193:37055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260926 20:23:21.494218 14762 catalog_manager.cc:4584] TS a410b1638b794d4b9f5475aee9fed5e1 (127.0.116.193:37055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e515f04999f249d3b628f4f2ea14a8af: Network error: Client connection negotiation failed: client connection to 127.0.116.193:37055: connect: Connection refused (error 111)
I20260926 20:23:21.500617 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 15106
I20260926 20:23:21.506659 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 14744
I20260926 20:23:21.513137 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:46601
--webserver_interface=127.0.116.254
--webserver_port=43999
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:46601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:23:21.627821 15271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:21.628109 15271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:21.628152 15271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:21.632424 15271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:23:21.632494 15271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:21.632514 15271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:21.632531 15271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:23:21.632591 15271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:23:21.637992 15271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:46601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:46601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=43999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:21.639189 15271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:21.640281 15271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:23:21.649252 15271 server_base.cc:1034] running on GCE node
W20260926 20:23:21.649223 15277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:21.649406 15276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:21.649299 15279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:21.650259 15271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:21.651566 15271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:21.652748 15271 hybrid_clock.cc:648] HybridClock initialized: now 1790454201652735 us; error 27 us; skew 500 ppm
Sep 26 20:23:21 dist-test-slave-qsm9 krb5kdc[14195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:21.655751 15271 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:23:21.656785 15271 webserver.cc:460] Webserver started at http://127.0.116.254:43999/ using document root <none> and password file <none>
I20260926 20:23:21.657387 15271 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:21.657454 15271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:21.661623 15271 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:23:21.664011 15286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:21.665372 15271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:23:21.665542 15271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "ff0e2eb98cb245028523b566380d48d7"
format_stamp: "Formatted at 2026-09-26 20:23:19 on dist-test-slave-qsm9"
server_key: "2f2fdd4a32dadad210078bb539135280"
server_key_iv: "437e56db8781c7ae4c2fcba32a58b57e"
server_key_version: "encryptionkey@0"
I20260926 20:23:21.666131 15271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:21.689443 15271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:21.693019 15271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:21.693249 15271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:21.700762 15271 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:46601
I20260926 20:23:21.701088 15339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:46601 every 8 connection(s)
I20260926 20:23:21.701941 15271 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:23:21.706342 15340 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:23:21.708039 15340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7: Bootstrap starting.
I20260926 20:23:21.710286 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 15271
I20260926 20:23:21.710777 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:37055
--local_ip_for_outbound_sockets=127.0.116.193
--tserver_master_addrs=127.0.116.254:46601
--webserver_port=34077
--webserver_interface=127.0.116.193
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:23:21.725725 15340 log.cc:824] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7: Log is configured to *not* fsync() on all Append() calls
I20260926 20:23:21.737318 15340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:23:21.737757 15340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7: Bootstrap complete.
I20260926 20:23:21.741966 15340 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0e2eb98cb245028523b566380d48d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46601 } }
I20260926 20:23:21.742419 15340 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff0e2eb98cb245028523b566380d48d7, State: Initialized, Role: FOLLOWER
I20260926 20:23:21.742987 15340 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0e2eb98cb245028523b566380d48d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46601 } }
I20260926 20:23:21.743147 15340 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:23:21.743235 15340 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:23:21.743352 15340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 2 FOLLOWER]: Advancing to term 3
I20260926 20:23:21.745488 15340 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0e2eb98cb245028523b566380d48d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46601 } }
I20260926 20:23:21.745883 15340 leader_election.cc:302] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff0e2eb98cb245028523b566380d48d7; no voters:
I20260926 20:23:21.746248 15340 leader_election.cc:288] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 20:23:21.746348 15344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 20:23:21.746560 15344 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [term 3 LEADER]: Becoming Leader. State: Replica: ff0e2eb98cb245028523b566380d48d7, State: Running, Role: LEADER
I20260926 20:23:21.746892 15344 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0e2eb98cb245028523b566380d48d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46601 } }
I20260926 20:23:21.747632 15340 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:23:21.747954 15344 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff0e2eb98cb245028523b566380d48d7. Latest consensus state: current_term: 3 leader_uuid: "ff0e2eb98cb245028523b566380d48d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0e2eb98cb245028523b566380d48d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46601 } } }
I20260926 20:23:21.748080 15344 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [sys.catalog]: This master's current role is: LEADER
I20260926 20:23:21.748492 15345 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ff0e2eb98cb245028523b566380d48d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0e2eb98cb245028523b566380d48d7" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46601 } } }
I20260926 20:23:21.748646 15348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:23:21.748672 15345 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff0e2eb98cb245028523b566380d48d7 [sys.catalog]: This master's current role is: LEADER
I20260926 20:23:21.754158 15348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:23:21.754925 15348 catalog_manager.cc:1259] Loaded cluster ID: 977edc0f6d874980b3c25b6b277af0fd
I20260926 20:23:21.755014 15348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 20:23:21.755987 15348 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe4722939ad google::LogMessage::Fail() at ??:0
@ 0x7fe47229688a google::LogMessage::SendToLog() at ??:0
@ 0x7fe4722934de google::LogMessage::Flush() at ??:0
@ 0x7fe472294a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe47cbd6fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fe47cbd7d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe47cbd4579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fe47cc18640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe4744e53d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe4744e5c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe4744e73f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe4744d6512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe475b1a6db start_thread at ??:0
@ 0x7fe470d9161f clone at ??:0
W20260926 20:23:21.857254 15342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:21.857528 15342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:21.857574 15342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:21.861781 15342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:21.861850 15342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:21.861912 15342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:23:21.867427 15342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:37055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=34077
--tserver_master_addrs=127.0.116.254:46601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:21.868608 15342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:21.869678 15342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:21.880398 15384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:21.880420 15382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:21.880456 15381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:21.880952 15342 server_base.cc:1034] running on GCE node
I20260926 20:23:21.881407 15342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:21.882074 15342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:21.883296 15342 hybrid_clock.cc:648] HybridClock initialized: now 1790454201883260 us; error 52 us; skew 500 ppm
Sep 26 20:23:21 dist-test-slave-qsm9 krb5kdc[14195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:21.893083 15342 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:23:21.894153 15342 webserver.cc:460] Webserver started at http://127.0.116.193:34077/ using document root <none> and password file <none>
I20260926 20:23:21.894745 15342 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:21.894835 15342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:21.899107 15342 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:23:21.901598 15391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:21.902923 15342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 20:23:21.903053 15342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "a410b1638b794d4b9f5475aee9fed5e1"
format_stamp: "Formatted at 2026-09-26 20:23:19 on dist-test-slave-qsm9"
server_key: "c3d457adf56dacf4d7656ebcb94a0e42"
server_key_iv: "d2f83fbdc0d1eb4b3d4a8299d0e99ebc"
server_key_version: "encryptionkey@0"
I20260926 20:23:21.903663 15342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:21.922847 15342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:21.926368 15342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:21.926582 15342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:21.927271 15342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:23:21.928517 15398 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:23:21.933987 15342 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:23:21.934103 15342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 20:23:21.934209 15342 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:23:21.936029 15342 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:23:21.936100 15342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260926 20:23:21.936411 15398 tablet_bootstrap.cc:492] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1: Bootstrap starting.
I20260926 20:23:21.948134 15342 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:37055
I20260926 20:23:21.948513 15505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:37055 every 8 connection(s)
I20260926 20:23:21.949316 15342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:23:21.958519 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 15342
I20260926 20:23:21.958983 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:37991
--local_ip_for_outbound_sockets=127.0.116.194
--tserver_master_addrs=127.0.116.254:46601
--webserver_port=33111
--webserver_interface=127.0.116.194
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:23:21.959225 15398 log.cc:824] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1: Log is configured to *not* fsync() on all Append() calls
W20260926 20:23:21.960052 15508 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:23:21.951455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.193:35659)
0926 20:23:21.952085 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:21.952145 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:21.953383 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:21.958936 (+ 5553us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":161,"threads_started":1}
W20260926 20:23:21.960587 15506 heartbeater.cc:644] Failed to heartbeat to 127.0.116.254:46601 (0 consecutive failures): Network error: Failed to ping master at 127.0.116.254:46601: Client connection negotiation failed: client connection to 127.0.116.254:46601: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:23:21.961370 15508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:21.961080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.193:41521)
0926 20:23:21.961220 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:21.961298 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 20:23:21.962335 15508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:21.962076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.193:54083)
0926 20:23:21.962200 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:21.962253 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 20:23:21.962556 15506 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:23:21.967561 15398 tablet_bootstrap.cc:492] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:23:21.967991 15398 tablet_bootstrap.cc:492] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1: Bootstrap complete.
I20260926 20:23:21.969220 15398 ts_tablet_manager.cc:1397] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1: Time spent bootstrapping tablet: real 0.033s user 0.030s sys 0.000s
I20260926 20:23:21.971940 15398 raft_consensus.cc:348] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } }
I20260926 20:23:21.972414 15398 raft_consensus.cc:729] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a410b1638b794d4b9f5475aee9fed5e1, State: Initialized, Role: FOLLOWER
I20260926 20:23:21.972918 15398 consensus_queue.cc:260] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } }
I20260926 20:23:21.974014 15398 ts_tablet_manager.cc:1428] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 20:23:21.974355 15508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:21.974000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.193:48337)
0926 20:23:21.974190 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:21.974260 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 20:23:21.975117 15508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:21.974885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.193:47421)
0926 20:23:21.975000 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:21.975055 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 20:23:22.073589 15510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:22.073874 15510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:22.073921 15510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:22.078094 15510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:22.078157 15510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:22.078215 15510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:23:22.083671 15510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:37991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=33111
--tserver_master_addrs=127.0.116.254:46601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:22.084746 15510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:22.085796 15510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:22.095896 15516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:22.096058 15517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:22.095896 15519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:22.096076 15510 server_base.cc:1034] running on GCE node
I20260926 20:23:22.096880 15510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:22.097555 15510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:22.098763 15510 hybrid_clock.cc:648] HybridClock initialized: now 1790454202098745 us; error 27 us; skew 500 ppm
Sep 26 20:23:22 dist-test-slave-qsm9 krb5kdc[14195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:22.101951 15510 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:23:22.102919 15510 webserver.cc:460] Webserver started at http://127.0.116.194:33111/ using document root <none> and password file <none>
I20260926 20:23:22.103590 15510 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:22.103679 15510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:22.107942 15510 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:23:22.109988 15526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:22.111284 15510 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:23:22.111433 15510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "1baa9bc83c5e423198f49cef556af54e"
format_stamp: "Formatted at 2026-09-26 20:23:19 on dist-test-slave-qsm9"
server_key: "40d8ec69c428090b1d549185bc50694b"
server_key_iv: "f484f6529cc4173b2088ba8c5c45026b"
server_key_version: "encryptionkey@0"
I20260926 20:23:22.112075 15510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:22.131942 15510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:22.135273 15510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:22.135525 15510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:22.136168 15510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:23:22.137038 15510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:23:22.137084 15510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:22.137126 15510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:23:22.137141 15510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:22.146265 15510 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:37991
I20260926 20:23:22.146287 15639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:37991 every 8 connection(s)
I20260926 20:23:22.147325 15510 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:23:22.151109 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:22.149534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.194:40095)
0926 20:23:22.150232 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:22.150350 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":116,"threads_started":1}
W20260926 20:23:22.151640 15640 heartbeater.cc:644] Failed to heartbeat to 127.0.116.254:46601 (0 consecutive failures): Network error: Failed to ping master at 127.0.116.254:46601: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
I20260926 20:23:22.152266 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:22.152039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.194:58905)
0926 20:23:22.152147 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:22.152209 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 20:23:22.152819 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:22.152635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.194:54409)
0926 20:23:22.152734 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:22.152774 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 20:23:22.152985 15640 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:23:22.156837 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 15510
I20260926 20:23:22.157222 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:39271
--local_ip_for_outbound_sockets=127.0.116.195
--tserver_master_addrs=127.0.116.254:46601
--webserver_port=40359
--webserver_interface=127.0.116.195
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:23:22.268997 15643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:22.269325 15643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:22.269407 15643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:22.273540 15643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:22.273635 15643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:22.273728 15643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:23:22.279078 15643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:39271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=40359
--tserver_master_addrs=127.0.116.254:46601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:22.280258 15643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:22.281311 15643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:22.291311 15649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:22.291314 15648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:22.291314 15651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:22.291371 15643 server_base.cc:1034] running on GCE node
I20260926 20:23:22.292495 15643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:22.293222 15643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:22.294385 15643 hybrid_clock.cc:648] HybridClock initialized: now 1790454202294365 us; error 32 us; skew 500 ppm
Sep 26 20:23:22 dist-test-slave-qsm9 krb5kdc[14195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:22.297390 15643 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:23:22.298363 15643 webserver.cc:460] Webserver started at http://127.0.116.195:40359/ using document root <none> and password file <none>
I20260926 20:23:22.298949 15643 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:22.299014 15643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:22.303166 15643 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:23:22.305444 15658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:22.306851 15643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 20:23:22.306990 15643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "f9aa02cf617040d78c5f86ce7f144961"
format_stamp: "Formatted at 2026-09-26 20:23:20 on dist-test-slave-qsm9"
server_key: "35dd14c426d0249f64d795ba41e0d4ea"
server_key_iv: "8a2e3bee538f3b0d9aa0772aa62177d2"
server_key_version: "encryptionkey@0"
I20260926 20:23:22.307705 15643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:22.319752 15643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:22.324033 15643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:22.324235 15643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:22.324872 15643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:23:22.325698 15643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:23:22.325765 15643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:22.325847 15643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:23:22.325896 15643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:22.334956 15643 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:39271
I20260926 20:23:22.334980 15771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:39271 every 8 connection(s)
I20260926 20:23:22.336009 15643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:23:22.339576 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:22.338045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.195:44721)
0926 20:23:22.338689 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:22.338813 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":113,"threads_started":1}
W20260926 20:23:22.339998 15772 heartbeater.cc:644] Failed to heartbeat to 127.0.116.254:46601 (0 consecutive failures): Network error: Failed to ping master at 127.0.116.254:46601: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
I20260926 20:23:22.340611 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:22.340393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.195:59967)
0926 20:23:22.340506 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:22.340556 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 20:23:22.341178 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:22.341010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.195:33397)
0926 20:23:22.341096 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:22.341135 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 20:23:22.341328 15772 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:23:22.343902 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 15643
I20260926 20:23:22.976327 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:22.975887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.193:41959)
0926 20:23:22.976144 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:22.976203 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260926 20:23:23.154143 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:23.153630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.194:47513)
0926 20:23:23.153965 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:23.154031 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":90,"threads_started":1}
I20260926 20:23:23.314558 15778 raft_consensus.cc:482] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:23:23.314811 15778 raft_consensus.cc:504] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } }
I20260926 20:23:23.316026 15778 leader_election.cc:288] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1baa9bc83c5e423198f49cef556af54e (127.0.116.194:37991), f9aa02cf617040d78c5f86ce7f144961 (127.0.116.195:39271)
Sep 26 20:23:23 dist-test-slave-qsm9 krb5kdc[14195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.195@KRBTEST.COM
Sep 26 20:23:23 dist-test-slave-qsm9 krb5kdc[14195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.194@KRBTEST.COM
I20260926 20:23:23.331308 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:23.316242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:39271 (local address 127.0.116.193:50717)
0926 20:23:23.316620 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:23.316634 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:23.316745 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:23.318320 (+ 1575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:23.318326 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:23.318884 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:23.319389 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:23.319404 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:23.320705 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:23.320709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:23.321637 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:23.321647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:23.322353 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:23.322357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:23.322554 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:23.323285 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:23.323304 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:23.325111 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:23.328015 (+ 2904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:23.328022 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:23.328037 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:23.328425 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:23.328921 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:23.328923 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:23.328926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:23.329158 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:23.330010 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:23.330017 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:23.330397 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 20:23:23.330688 (+ 291us) client_negotiation.cc:240] Negotiation successful
0926 20:23:23.331124 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":312,"spinlock_wait_cycles":60416,"thread_start_us":97,"threads_started":1}
I20260926 20:23:23.331352 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:23.316429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:50717 (local address 127.0.116.195:39271)
0926 20:23:23.316786 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:23.316800 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:23.316845 (+ 45us) server_negotiation.cc:407] Connection header received
0926 20:23:23.317574 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:23.317580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:23.317952 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:23.318259 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:23.319586 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:23.320551 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:23.321769 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:23.322223 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:23.322407 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:23.325401 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:23.325409 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:23.325427 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:23.325465 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:23.327866 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:23.328554 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:23.328578 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:23.328589 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:23.328830 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:23.329264 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:23.329267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:23.329270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:23.329875 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:23.330017 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:23.330833 (+ 816us) server_negotiation.cc:299] Negotiation successful
0926 20:23:23.331141 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":60,"threads_started":1}
I20260926 20:23:23.331363 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:23.316399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:47513 (local address 127.0.116.194:37991)
0926 20:23:23.316871 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:23.316890 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:23.316922 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:23:23.317614 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:23.317620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:23.317952 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:23.318259 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:23.319595 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:23.320553 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:23.321761 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:23.322223 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:23.322405 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:23.325800 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:23.325811 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:23.325832 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:23.325875 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:23.328157 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:23.328624 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:23.328632 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:23.328640 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:23.328797 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:23.329264 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:23.329267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:23.329269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:23.329854 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:23.330009 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:23.330814 (+ 805us) server_negotiation.cc:299] Negotiation successful
0926 20:23:23.331141 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":159,"threads_started":1}
I20260926 20:23:23.331310 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:23.316180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:37991 (local address 127.0.116.193:47513)
0926 20:23:23.316377 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:23.316394 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:23.316523 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:23.318321 (+ 1798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:23.318327 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:23.318885 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:23.319389 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:23.319404 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:23.320693 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:23.320701 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:23.321639 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:23.321648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:23.322353 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:23.322357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:23.322576 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:23.323284 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:23.323306 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:23.325555 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:23.328271 (+ 2716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:23.328277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:23.328283 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:23.328522 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:23.328920 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:23.328923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:23.328926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:23.329162 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:23.330009 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:23.330017 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:23.330397 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 20:23:23.330641 (+ 244us) client_negotiation.cc:240] Negotiation successful
0926 20:23:23.331069 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":223,"spinlock_wait_cycles":56192}
W20260926 20:23:23.334255 15393 leader_election.cc:341] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9aa02cf617040d78c5f86ce7f144961 (127.0.116.195:39271): Not found: Tablet not found: e515f04999f249d3b628f4f2ea14a8af
W20260926 20:23:23.334431 15394 leader_election.cc:341] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1baa9bc83c5e423198f49cef556af54e (127.0.116.194:37991): Not found: Tablet not found: e515f04999f249d3b628f4f2ea14a8af
I20260926 20:23:23.334540 15394 leader_election.cc:302] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a410b1638b794d4b9f5475aee9fed5e1; no voters: 1baa9bc83c5e423198f49cef556af54e, f9aa02cf617040d78c5f86ce7f144961
I20260926 20:23:23.334853 15778 raft_consensus.cc:2726] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:23:23.342381 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:23.341935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.195:36399)
0926 20:23:23.342211 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:23.342281 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":67,"threads_started":1}
I20260926 20:23:23.977869 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:23.977352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.193:36931)
0926 20:23:23.977693 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:23.977766 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":112,"threads_started":1}
I20260926 20:23:24.155369 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:24.154935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.194:49261)
0926 20:23:24.155210 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:24.155271 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":74,"threads_started":1}
I20260926 20:23:24.343751 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:24.343240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.195:37677)
0926 20:23:24.343565 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:24.343634 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
I20260926 20:23:24.979328 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:24.978775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.193:53073)
0926 20:23:24.979139 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:24.979216 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":100,"threads_started":1}
I20260926 20:23:25.017400 15787 raft_consensus.cc:482] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:23:25.017526 15787 raft_consensus.cc:504] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } }
I20260926 20:23:25.017982 15787 leader_election.cc:288] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1baa9bc83c5e423198f49cef556af54e (127.0.116.194:37991), f9aa02cf617040d78c5f86ce7f144961 (127.0.116.195:39271)
W20260926 20:23:25.018762 15393 leader_election.cc:341] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9aa02cf617040d78c5f86ce7f144961 (127.0.116.195:39271): Not found: Tablet not found: e515f04999f249d3b628f4f2ea14a8af
W20260926 20:23:25.018872 15394 leader_election.cc:341] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1baa9bc83c5e423198f49cef556af54e (127.0.116.194:37991): Not found: Tablet not found: e515f04999f249d3b628f4f2ea14a8af
I20260926 20:23:25.018985 15394 leader_election.cc:302] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a410b1638b794d4b9f5475aee9fed5e1; no voters: 1baa9bc83c5e423198f49cef556af54e, f9aa02cf617040d78c5f86ce7f144961
I20260926 20:23:25.019174 15787 raft_consensus.cc:2726] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:23:25.156646 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:25.156200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.194:60147)
0926 20:23:25.156477 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:25.156547 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":76,"threads_started":1}
I20260926 20:23:25.345096 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:25.344664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.195:45669)
0926 20:23:25.344931 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:25.344998 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":77,"threads_started":1}
I20260926 20:23:25.980676 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:25.980191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.193:53141)
0926 20:23:25.980494 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:25.980563 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":77,"threads_started":1}
I20260926 20:23:26.158006 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:26.157513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.194:35079)
0926 20:23:26.157834 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:26.157903 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":82,"threads_started":1}
I20260926 20:23:26.346318 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:26.345907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.195:48583)
0926 20:23:26.346164 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:26.346222 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":77,"threads_started":1}
I20260926 20:23:26.982350 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:26.981816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.193:54441)
0926 20:23:26.982155 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:26.982237 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":124,"threads_started":1}
I20260926 20:23:27.159149 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:27.158743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.194:46799)
0926 20:23:27.158994 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:27.159049 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
I20260926 20:23:27.347564 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:27.347048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.195:44201)
0926 20:23:27.347326 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:27.347398 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":70,"threads_started":1}
I20260926 20:23:27.422865 15796 raft_consensus.cc:482] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:23:27.423027 15796 raft_consensus.cc:504] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } }
I20260926 20:23:27.423519 15796 leader_election.cc:288] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1baa9bc83c5e423198f49cef556af54e (127.0.116.194:37991), f9aa02cf617040d78c5f86ce7f144961 (127.0.116.195:39271)
W20260926 20:23:27.424245 15393 leader_election.cc:341] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9aa02cf617040d78c5f86ce7f144961 (127.0.116.195:39271): Not found: Tablet not found: e515f04999f249d3b628f4f2ea14a8af
W20260926 20:23:27.424412 15394 leader_election.cc:341] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1baa9bc83c5e423198f49cef556af54e (127.0.116.194:37991): Not found: Tablet not found: e515f04999f249d3b628f4f2ea14a8af
I20260926 20:23:27.424475 15394 leader_election.cc:302] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a410b1638b794d4b9f5475aee9fed5e1; no voters: 1baa9bc83c5e423198f49cef556af54e, f9aa02cf617040d78c5f86ce7f144961
I20260926 20:23:27.424610 15796 raft_consensus.cc:2726] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:23:27.983878 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:27.983191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.193:49153)
0926 20:23:27.983658 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:27.983771 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":86,"threads_started":1}
I20260926 20:23:28.160423 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:28.159987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.194:57873)
0926 20:23:28.160263 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:28.160325 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":90,"threads_started":1}
I20260926 20:23:28.348836 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:28.348422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.195:33939)
0926 20:23:28.348669 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:28.348740 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260926 20:23:28.985169 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:28.984703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.193:49763)
0926 20:23:28.985007 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:28.985068 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":126,"threads_started":1}
I20260926 20:23:29.161831 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:29.161209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.194:34131)
0926 20:23:29.161584 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:29.161707 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":90,"threads_started":1}
I20260926 20:23:29.350181 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:29.349657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.195:35369)
0926 20:23:29.350017 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:29.350082 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":82,"threads_started":1}
I20260926 20:23:29.986518 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:29.986047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.193:36721)
0926 20:23:29.986354 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:29.986420 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":94,"threads_started":1}
I20260926 20:23:30.163172 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:30.162728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.194:35943)
0926 20:23:30.163008 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:30.163073 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":86,"threads_started":1}
I20260926 20:23:30.351642 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:30.351108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.195:43203)
0926 20:23:30.351385 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:30.351458 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
I20260926 20:23:30.852043 15806 raft_consensus.cc:482] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:23:30.852177 15806 raft_consensus.cc:504] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } }
I20260926 20:23:30.852802 15806 leader_election.cc:288] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1baa9bc83c5e423198f49cef556af54e (127.0.116.194:37991), f9aa02cf617040d78c5f86ce7f144961 (127.0.116.195:39271)
W20260926 20:23:30.853559 15393 leader_election.cc:341] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9aa02cf617040d78c5f86ce7f144961 (127.0.116.195:39271): Not found: Tablet not found: e515f04999f249d3b628f4f2ea14a8af
W20260926 20:23:30.853653 15394 leader_election.cc:341] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1baa9bc83c5e423198f49cef556af54e (127.0.116.194:37991): Not found: Tablet not found: e515f04999f249d3b628f4f2ea14a8af
I20260926 20:23:30.853710 15394 leader_election.cc:302] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a410b1638b794d4b9f5475aee9fed5e1; no voters: 1baa9bc83c5e423198f49cef556af54e, f9aa02cf617040d78c5f86ce7f144961
I20260926 20:23:30.853863 15806 raft_consensus.cc:2726] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:23:30.988055 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:30.987474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.193:55145)
0926 20:23:30.987883 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:30.987952 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":132,"threads_started":1}
I20260926 20:23:31.164548 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:31.164040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.194:47455)
0926 20:23:31.164359 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:31.164440 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":98,"threads_started":1}
I20260926 20:23:31.353122 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:31.352619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.195:53235)
0926 20:23:31.352929 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:31.352985 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":89,"threads_started":1}
I20260926 20:23:31.989488 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:31.989074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.193:54763)
0926 20:23:31.989327 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:31.989389 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":77,"threads_started":1}
I20260926 20:23:32.165886 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:32.165443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.194:57933)
0926 20:23:32.165724 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:32.165788 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
I20260926 20:23:32.354375 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:32.353960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.195:34449)
0926 20:23:32.354209 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:32.354274 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":74,"threads_started":1}
I20260926 20:23:32.990954 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:32.990465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.193:59607)
0926 20:23:32.990788 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:32.990856 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":105,"threads_started":1}
I20260926 20:23:33.167398 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:33.166919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.194:44507)
0926 20:23:33.167211 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:33.167287 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":79,"threads_started":1}
I20260926 20:23:33.355912 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:33.355325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.195:52141)
0926 20:23:33.355741 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:33.355812 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":81,"threads_started":1}
I20260926 20:23:33.992358 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:33.991914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.193:35747)
0926 20:23:33.992186 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:33.992256 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":79,"threads_started":1}
I20260926 20:23:34.168692 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:34.168233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.194:51573)
0926 20:23:34.168538 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:34.168596 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":81,"threads_started":1}
I20260926 20:23:34.357295 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:34.356865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.195:41045)
0926 20:23:34.357134 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:34.357190 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":102,"threads_started":1}
I20260926 20:23:34.993691 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:34.993212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.193:53929)
0926 20:23:34.993521 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:34.993590 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":94,"threads_started":1}
I20260926 20:23:35.169993 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:35.169539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.194:51457)
0926 20:23:35.169833 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:35.169896 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":72,"threads_started":1}
I20260926 20:23:35.358595 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:35.358150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.195:53255)
0926 20:23:35.358431 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:35.358496 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":109,"threads_started":1}
I20260926 20:23:35.995110 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:35.994648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.193:37543)
0926 20:23:35.994944 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:35.995011 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":87,"threads_started":1}
I20260926 20:23:36.171324 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:36.170810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.194:40975)
0926 20:23:36.171120 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:36.171199 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":96,"threads_started":1}
I20260926 20:23:36.360117 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:36.359652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.195:39241)
0926 20:23:36.359925 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:36.359991 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":74,"threads_started":1}
I20260926 20:23:36.996582 15825 raft_consensus.cc:482] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:23:36.996721 15825 raft_consensus.cc:504] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1baa9bc83c5e423198f49cef556af54e" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 37991 } } peers { permanent_uuid: "f9aa02cf617040d78c5f86ce7f144961" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 39271 } } peers { permanent_uuid: "a410b1638b794d4b9f5475aee9fed5e1" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 37055 } }
I20260926 20:23:36.997025 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:36.996363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.193:41319)
0926 20:23:36.996818 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:36.996920 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":67,"threads_started":1}
I20260926 20:23:36.997305 15825 leader_election.cc:288] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1baa9bc83c5e423198f49cef556af54e (127.0.116.194:37991), f9aa02cf617040d78c5f86ce7f144961 (127.0.116.195:39271)
W20260926 20:23:36.998080 15394 leader_election.cc:341] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1baa9bc83c5e423198f49cef556af54e (127.0.116.194:37991): Not found: Tablet not found: e515f04999f249d3b628f4f2ea14a8af
W20260926 20:23:36.998183 15393 leader_election.cc:341] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9aa02cf617040d78c5f86ce7f144961 (127.0.116.195:39271): Not found: Tablet not found: e515f04999f249d3b628f4f2ea14a8af
I20260926 20:23:36.998241 15393 leader_election.cc:302] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a410b1638b794d4b9f5475aee9fed5e1; no voters: 1baa9bc83c5e423198f49cef556af54e, f9aa02cf617040d78c5f86ce7f144961
I20260926 20:23:36.998391 15825 raft_consensus.cc:2726] T e515f04999f249d3b628f4f2ea14a8af P a410b1638b794d4b9f5475aee9fed5e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:23:37.172823 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:37.172349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.194:35795)
0926 20:23:37.172642 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:37.172709 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":75,"threads_started":1}
I20260926 20:23:37.356170 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 15342
I20260926 20:23:37.361656 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:37.361231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46601 (local address 127.0.116.195:55759)
0926 20:23:37.361497 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:37.361560 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":97,"threads_started":1}
I20260926 20:23:37.362303 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 15510
I20260926 20:23:37.368250 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 15643
2026-09-26T20:23:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19478 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:23:37 dist-test-slave-qsm9 krb5kdc[15831](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:23:37 dist-test-slave-qsm9 krb5kdc[15831](info): set up 2 sockets
Sep 26 20:23:37 dist-test-slave-qsm9 krb5kdc[15831](info): commencing operation
krb5kdc: starting...
W20260926 20:23:38.616086 467 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:23:38 dist-test-slave-qsm9 krb5kdc[15831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:23:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:23:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:23:38.742336 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:45107
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:35573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:45107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:23:38.854527 15844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:38.854809 15844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:38.854851 15844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:38.858872 15844 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 20:23:38.858927 15844 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 20:23:38.858942 15844 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 20:23:38.858958 15844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:23:38.858976 15844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:38.859043 15844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:38.859074 15844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:23:38.859109 15844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:23:38.864321 15844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:35573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:45107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:45107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:38.865435 15844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:38.866505 15844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:38.875648 15850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:38.875677 15852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:38.875677 15849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:38.875790 15844 server_base.cc:1034] running on GCE node
I20260926 20:23:38.876487 15844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:38.877619 15844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:38.878803 15844 hybrid_clock.cc:648] HybridClock initialized: now 1790454218878770 us; error 47 us; skew 500 ppm
Sep 26 20:23:38 dist-test-slave-qsm9 krb5kdc[15831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:38.881896 15844 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:23:38.882884 15844 webserver.cc:460] Webserver started at http://127.0.116.254:43421/ using document root <none> and password file <none>
I20260926 20:23:38.883479 15844 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:38.883602 15844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:38.883843 15844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:23:38.886006 15844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "33516122d3ae4d59a744fd889e0b7400"
format_stamp: "Formatted at 2026-09-26 20:23:38 on dist-test-slave-qsm9"
server_key: "8079862cee6a536c39c9f0db86e727fe"
server_key_iv: "201831c6e49559e059fbc052f1f05882"
server_key_version: "encryptionkey@0"
I20260926 20:23:38.886617 15844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "33516122d3ae4d59a744fd889e0b7400"
format_stamp: "Formatted at 2026-09-26 20:23:38 on dist-test-slave-qsm9"
server_key: "8079862cee6a536c39c9f0db86e727fe"
server_key_iv: "201831c6e49559e059fbc052f1f05882"
server_key_version: "encryptionkey@0"
I20260926 20:23:38.890821 15844 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 20:23:38.893561 15859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:38.894964 15844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:23:38.895133 15844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "33516122d3ae4d59a744fd889e0b7400"
format_stamp: "Formatted at 2026-09-26 20:23:38 on dist-test-slave-qsm9"
server_key: "8079862cee6a536c39c9f0db86e727fe"
server_key_iv: "201831c6e49559e059fbc052f1f05882"
server_key_version: "encryptionkey@0"
I20260926 20:23:38.895264 15844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:38.913882 15844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:38.917454 15844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:38.917685 15844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:38.925244 15844 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:45107
I20260926 20:23:38.925536 15911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:45107 every 8 connection(s)
I20260926 20:23:38.926316 15844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:23:38.929270 15912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:23:38.929346 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 15844
I20260926 20:23:38.929520 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:23:38.929814 467 external_mini_cluster.cc:1428] Setting key aa53ac06c440794613e3daf1accd0dd4
I20260926 20:23:38.935902 15912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400: Bootstrap starting.
I20260926 20:23:38.938294 15912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400: Neither blocks nor log segments found. Creating new log.
Sep 26 20:23:38 dist-test-slave-qsm9 krb5kdc[15831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:23:38.939363 15912 log.cc:824] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400: Log is configured to *not* fsync() on all Append() calls
I20260926 20:23:38.941885 15912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400: No bootstrap required, opened a new log
I20260926 20:23:38.945349 15915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:38.931154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57074 (local address 127.0.116.254:45107)
0926 20:23:38.931621 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:38.931642 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:38.931868 (+ 226us) server_negotiation.cc:407] Connection header received
0926 20:23:38.932740 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:38.932759 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:38.933098 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:38.933442 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:38.934301 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:38.935256 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:38.936255 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:38.936729 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:38.936914 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:38.939756 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:38.939762 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:38.939777 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:38.939807 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:38.941801 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:38.942395 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:38.942401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:38.942407 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:38.942552 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:38.942890 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:38.942893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:38.942895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:38.943389 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:38.943619 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:38.943958 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 20:23:38.944300 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":154,"threads_started":1}
I20260926 20:23:38.946707 15912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33516122d3ae4d59a744fd889e0b7400" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45107 } }
I20260926 20:23:38.946929 15912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:23:38.946964 15912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33516122d3ae4d59a744fd889e0b7400, State: Initialized, Role: FOLLOWER
I20260926 20:23:38.947449 15912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33516122d3ae4d59a744fd889e0b7400" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45107 } }
I20260926 20:23:38.947664 15912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:23:38.947733 15912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:23:38.947830 15912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:23:38.948783 15912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33516122d3ae4d59a744fd889e0b7400" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45107 } }
I20260926 20:23:38.949159 15912 leader_election.cc:302] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33516122d3ae4d59a744fd889e0b7400; no voters:
I20260926 20:23:38.949506 15912 leader_election.cc:288] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:23:38.949645 15917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:23:38.949906 15917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 1 LEADER]: Becoming Leader. State: Replica: 33516122d3ae4d59a744fd889e0b7400, State: Running, Role: LEADER
I20260926 20:23:38.950217 15917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33516122d3ae4d59a744fd889e0b7400" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45107 } }
I20260926 20:23:38.950840 15912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:23:38.952131 15919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33516122d3ae4d59a744fd889e0b7400. Latest consensus state: current_term: 1 leader_uuid: "33516122d3ae4d59a744fd889e0b7400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33516122d3ae4d59a744fd889e0b7400" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45107 } } }
I20260926 20:23:38.952085 15918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33516122d3ae4d59a744fd889e0b7400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33516122d3ae4d59a744fd889e0b7400" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45107 } } }
I20260926 20:23:38.952241 15919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [sys.catalog]: This master's current role is: LEADER
I20260926 20:23:38.952243 15918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [sys.catalog]: This master's current role is: LEADER
I20260926 20:23:38.952648 15926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:23:38.955828 15926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:23:38.961764 15926 catalog_manager.cc:1347] Generated new cluster ID: 22541c5ff5604266b729cd3f8379f1a0
I20260926 20:23:38.962123 15926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:23:38.983291 15926 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:23:38.984414 15926 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:23:38.992029 15926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400: Generated new TSK 0
I20260926 20:23:38.992936 15926 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
I20260926 20:23:39.027086 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:45107
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:35573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:23:39.142623 15938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:39.142926 15938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:39.143019 15938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:39.147886 15938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:39.147961 15938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:39.148059 15938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:23:39.153878 15938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:35573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:45107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:39.155090 15938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:39.156219 15938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:39.166570 15946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:39.166607 15943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:39.166608 15938 server_base.cc:1034] running on GCE node
W20260926 20:23:39.166591 15944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:39.167328 15938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:39.168093 15938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:39.169260 15938 hybrid_clock.cc:648] HybridClock initialized: now 1790454219169238 us; error 62 us; skew 500 ppm
Sep 26 20:23:39 dist-test-slave-qsm9 krb5kdc[15831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:39.172221 15938 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:23:39.173200 15938 webserver.cc:460] Webserver started at http://127.0.116.193:45871/ using document root <none> and password file <none>
I20260926 20:23:39.173790 15938 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:39.173858 15938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:39.174039 15938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:23:39.176360 15938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "e77b5a0d96c94986a49ba05cc9a068fe"
format_stamp: "Formatted at 2026-09-26 20:23:39 on dist-test-slave-qsm9"
server_key: "405981b7f7342f311bee0fc3ca5d9d7b"
server_key_iv: "1d847ab139783c77d5e7332ca1e1447f"
server_key_version: "encryptionkey@0"
I20260926 20:23:39.176935 15938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "e77b5a0d96c94986a49ba05cc9a068fe"
format_stamp: "Formatted at 2026-09-26 20:23:39 on dist-test-slave-qsm9"
server_key: "405981b7f7342f311bee0fc3ca5d9d7b"
server_key_iv: "1d847ab139783c77d5e7332ca1e1447f"
server_key_version: "encryptionkey@0"
I20260926 20:23:39.181190 15938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:23:39.183952 15953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:39.185328 15938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:23:39.185469 15938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "e77b5a0d96c94986a49ba05cc9a068fe"
format_stamp: "Formatted at 2026-09-26 20:23:39 on dist-test-slave-qsm9"
server_key: "405981b7f7342f311bee0fc3ca5d9d7b"
server_key_iv: "1d847ab139783c77d5e7332ca1e1447f"
server_key_version: "encryptionkey@0"
I20260926 20:23:39.185593 15938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:39.214365 15938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:39.218333 15938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:39.218578 15938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:39.219251 15938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:23:39.220197 15938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:23:39.220244 15938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:39.220309 15938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:23:39.220340 15938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:39.230166 15938 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:43847
I20260926 20:23:39.230244 16066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:43847 every 8 connection(s)
I20260926 20:23:39.231194 15938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:23:39.234193 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 15938
I20260926 20:23:39.234341 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:23:39.234732 467 external_mini_cluster.cc:1428] Setting key 6a73ab9ddd1e051b31c425e9e077b751
Sep 26 20:23:39 dist-test-slave-qsm9 krb5kdc[15831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.116.194@KRBTEST.COM; defaulting to no policy
I20260926 20:23:39.249537 15915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:39.233446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:35813 (local address 127.0.116.254:45107)
0926 20:23:39.233619 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:39.233622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:39.234588 (+ 966us) server_negotiation.cc:407] Connection header received
0926 20:23:39.236372 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:39.236375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:39.236424 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:39.236511 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:39.238089 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.239111 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.241380 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.241667 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.241781 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:39.245289 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:39.245292 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:39.245294 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:39.245324 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:39.247103 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:39.247822 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:39.247826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:39.247827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:39.247907 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:39.248264 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:39.248268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:39.248269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:39.248483 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:39.248587 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:39.249202 (+ 615us) server_negotiation.cc:299] Negotiation successful
0926 20:23:39.249354 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.194@KRBTEST.COM" created.
I20260926 20:23:39.250531 16069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:39.233852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.193:35813)
0926 20:23:39.234388 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:39.234449 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:39.236102 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:39.236783 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:39.236796 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:39.237305 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:39.237877 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:39.237894 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.239308 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.239313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:39.241185 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:39.241204 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.241899 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.241908 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:39.242190 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:39.243010 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:39.243046 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:39.245062 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:39.247272 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:39.247279 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:39.247297 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:39.247658 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:39.248004 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:39.248007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:39.248009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:39.248157 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:39.248586 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:39.248593 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:39.248888 (+ 295us) client_negotiation.cc:769] Sending connection context
0926 20:23:39.249140 (+ 252us) client_negotiation.cc:240] Negotiation successful
0926 20:23:39.249585 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":166,"threads_started":1}
I20260926 20:23:39.252014 16067 heartbeater.cc:347] Connected to a master server at 127.0.116.254:45107
I20260926 20:23:39.252313 16067 heartbeater.cc:464] Registering TS with master...
I20260926 20:23:39.253065 16067 heartbeater.cc:511] Master 127.0.116.254:45107 requested a full tablet report, sending...
I20260926 20:23:39.254942 15876 ts_manager.cc:194] Registered new tserver with Master: e77b5a0d96c94986a49ba05cc9a068fe (127.0.116.193:43847)
I20260926 20:23:39.256949 15876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:35813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
I20260926 20:23:39.270475 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:45107
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:35573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:23:39.384369 16072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:39.384652 16072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:39.384704 16072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:39.388741 16072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:39.388803 16072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:39.388864 16072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:23:39.395572 16072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:35573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:45107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:39.397028 16072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:39.398474 16072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:39.409440 16080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:39.409440 16078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:39.409456 16077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:39.409731 16072 server_base.cc:1034] running on GCE node
I20260926 20:23:39.410121 16072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:39.410779 16072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:39.411967 16072 hybrid_clock.cc:648] HybridClock initialized: now 1790454219411950 us; error 33 us; skew 500 ppm
Sep 26 20:23:39 dist-test-slave-qsm9 krb5kdc[15831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:39.414755 16072 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:23:39.415740 16072 webserver.cc:460] Webserver started at http://127.0.116.194:39795/ using document root <none> and password file <none>
I20260926 20:23:39.416323 16072 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:39.416448 16072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:39.416679 16072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:23:39.418748 16072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "b90889308e1941e28b5eb97e45b1f913"
format_stamp: "Formatted at 2026-09-26 20:23:39 on dist-test-slave-qsm9"
server_key: "3b410352a9c77998c7068e763a51a835"
server_key_iv: "40839f8e2c7c061971a06872c8ce749d"
server_key_version: "encryptionkey@0"
I20260926 20:23:39.419322 16072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "b90889308e1941e28b5eb97e45b1f913"
format_stamp: "Formatted at 2026-09-26 20:23:39 on dist-test-slave-qsm9"
server_key: "3b410352a9c77998c7068e763a51a835"
server_key_iv: "40839f8e2c7c061971a06872c8ce749d"
server_key_version: "encryptionkey@0"
I20260926 20:23:39.423161 16072 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260926 20:23:39.425714 16087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:39.426999 16072 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:23:39.427130 16072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "b90889308e1941e28b5eb97e45b1f913"
format_stamp: "Formatted at 2026-09-26 20:23:39 on dist-test-slave-qsm9"
server_key: "3b410352a9c77998c7068e763a51a835"
server_key_iv: "40839f8e2c7c061971a06872c8ce749d"
server_key_version: "encryptionkey@0"
I20260926 20:23:39.427243 16072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:39.447146 16072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:39.450311 16072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:39.450549 16072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:39.451227 16072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:23:39.452255 16072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:23:39.452322 16072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:39.452401 16072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:23:39.452443 16072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:39.461678 16072 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:42889
I20260926 20:23:39.461696 16200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:42889 every 8 connection(s)
I20260926 20:23:39.462601 16072 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:23:39.467557 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 16072
I20260926 20:23:39.467679 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:23:39.467974 467 external_mini_cluster.cc:1428] Setting key 116b297883ed53b2ed2ca45c107b821f
Sep 26 20:23:39 dist-test-slave-qsm9 krb5kdc[15831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:23:39.477073 15915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:39.464441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:36127 (local address 127.0.116.254:45107)
0926 20:23:39.464594 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:39.464598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:39.465464 (+ 866us) server_negotiation.cc:407] Connection header received
0926 20:23:39.466832 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:39.466835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:39.466932 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:39.467042 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:39.468462 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.469195 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.470342 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.470573 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.470688 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:39.473480 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:39.473483 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:39.473485 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:39.473509 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:39.474996 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:39.475560 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:39.475568 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:39.475569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:39.475641 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:39.475983 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:39.475987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:39.475989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:39.476172 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:39.476275 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:39.476773 (+ 498us) server_negotiation.cc:299] Negotiation successful
0926 20:23:39.476907 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":64}
WARNING: no policy specified for kudu/127.0.116.195@KRBTEST.COM; defaulting to no policy
I20260926 20:23:39.477903 16203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:39.464840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.194:36127)
0926 20:23:39.465286 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:39.465329 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:39.466601 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:39.467223 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:39.467230 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:39.467721 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:39.468292 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:39.468306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.469306 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.469309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:39.470199 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:39.470207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.470729 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.470733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:39.470914 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:39.471600 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:39.471618 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:39.473299 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:39.475132 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:39.475139 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:39.475156 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:39.475418 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:39.475734 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:39.475738 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:39.475740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:39.475887 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:39.476273 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:39.476280 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:39.476536 (+ 256us) client_negotiation.cc:769] Sending connection context
0926 20:23:39.476734 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 20:23:39.477081 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":116,"threads_started":1}
I20260926 20:23:39.479207 16201 heartbeater.cc:347] Connected to a master server at 127.0.116.254:45107
I20260926 20:23:39.479521 16201 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.195@KRBTEST.COM" created.
I20260926 20:23:39.480146 16201 heartbeater.cc:511] Master 127.0.116.254:45107 requested a full tablet report, sending...
I20260926 20:23:39.481295 15876 ts_manager.cc:194] Registered new tserver with Master: b90889308e1941e28b5eb97e45b1f913 (127.0.116.194:42889)
I20260926 20:23:39.481968 15876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:36127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
I20260926 20:23:39.498507 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:45107
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:35573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:23:39.615577 16206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:23:39.615860 16206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:23:39.615939 16206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:23:39.620044 16206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:23:39.620107 16206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:23:39.620168 16206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:23:39.625536 16206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:35573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:45107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:23:39.626602 16206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:23:39.627655 16206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:23:39.637394 16211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:23:39.637403 16212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:39.637377 16206 server_base.cc:1034] running on GCE node
W20260926 20:23:39.637436 16214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:23:39.638027 16206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:23:39.638701 16206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:23:39.639863 16206 hybrid_clock.cc:648] HybridClock initialized: now 1790454219639846 us; error 29 us; skew 500 ppm
Sep 26 20:23:39 dist-test-slave-qsm9 krb5kdc[15831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:23:39.642705 16206 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:23:39.643649 16206 webserver.cc:460] Webserver started at http://127.0.116.195:34049/ using document root <none> and password file <none>
I20260926 20:23:39.644218 16206 fs_manager.cc:359] Metadata directory not provided
I20260926 20:23:39.644279 16206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:23:39.644501 16206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:23:39.646552 16206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "bc16bee212ed4616bbf66ff7dc175799"
format_stamp: "Formatted at 2026-09-26 20:23:39 on dist-test-slave-qsm9"
server_key: "69b2813d14c632061e625ad8e4c78a8c"
server_key_iv: "dc992dca010579bfc27c679a75f94d49"
server_key_version: "encryptionkey@0"
I20260926 20:23:39.647133 16206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "bc16bee212ed4616bbf66ff7dc175799"
format_stamp: "Formatted at 2026-09-26 20:23:39 on dist-test-slave-qsm9"
server_key: "69b2813d14c632061e625ad8e4c78a8c"
server_key_iv: "dc992dca010579bfc27c679a75f94d49"
server_key_version: "encryptionkey@0"
I20260926 20:23:39.651109 16206 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 20:23:39.653951 16221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:39.655287 16206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 20:23:39.655436 16206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "bc16bee212ed4616bbf66ff7dc175799"
format_stamp: "Formatted at 2026-09-26 20:23:39 on dist-test-slave-qsm9"
server_key: "69b2813d14c632061e625ad8e4c78a8c"
server_key_iv: "dc992dca010579bfc27c679a75f94d49"
server_key_version: "encryptionkey@0"
I20260926 20:23:39.655604 16206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:23:39.673619 16206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:23:39.677006 16206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:23:39.677218 16206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:23:39.677811 16206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:23:39.678669 16206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:23:39.678715 16206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:39.678754 16206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:23:39.678769 16206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:23:39.688025 16206 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:38389
I20260926 20:23:39.688114 16335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:38389 every 8 connection(s)
I20260926 20:23:39.689086 16206 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:23:39.695104 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 16206
I20260926 20:23:39.695230 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:23:39.695564 467 external_mini_cluster.cc:1428] Setting key 4398ab173eec182c344870f2ceeda0a6
Sep 26 20:23:39 dist-test-slave-qsm9 krb5kdc[15831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:23:39.703696 15915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:39.691129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:59467 (local address 127.0.116.254:45107)
0926 20:23:39.691278 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:39.691281 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:39.692148 (+ 867us) server_negotiation.cc:407] Connection header received
0926 20:23:39.693440 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:39.693443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:39.693494 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:39.693571 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:39.694832 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.695673 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.697213 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.697462 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.697546 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:39.700183 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:39.700187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:39.700189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:39.700215 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:39.701708 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:39.702223 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:39.702225 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:39.702226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:39.702299 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:39.702642 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:39.702644 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:39.702645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:39.702816 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:39.702891 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:39.703416 (+ 525us) server_negotiation.cc:299] Negotiation successful
0926 20:23:39.703574 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:23:39.704629 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:39.691434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.195:59467)
0926 20:23:39.691983 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:39.692030 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:39.693231 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:39.693750 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:39.693758 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:39.694170 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:23:39.694666 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:39.694680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.695816 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.695823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:39.697083 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:39.697095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.697618 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.697622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:39.697822 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:39.698435 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:23:39.698452 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:23:39.700014 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:23:39.701834 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:39.701839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:39.701852 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:39.702128 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:39.702403 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:23:39.702406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:23:39.702407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:23:39.702555 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:23:39.702914 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:23:39.702919 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:23:39.703179 (+ 260us) client_negotiation.cc:769] Sending connection context
0926 20:23:39.703362 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 20:23:39.703710 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":137,"threads_started":1}
I20260926 20:23:39.705869 16336 heartbeater.cc:347] Connected to a master server at 127.0.116.254:45107
I20260926 20:23:39.706163 16336 heartbeater.cc:464] Registering TS with master...
I20260926 20:23:39.706696 16336 heartbeater.cc:511] Master 127.0.116.254:45107 requested a full tablet report, sending...
I20260926 20:23:39.707903 15876 ts_manager.cc:194] Registered new tserver with Master: bc16bee212ed4616bbf66ff7dc175799 (127.0.116.195:38389)
I20260926 20:23:39.708541 15876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:59467
I20260926 20:23:39.710120 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:23:39.720314 15915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:39.712348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57090 (local address 127.0.116.254:45107)
0926 20:23:39.712501 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:39.712505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:39.712689 (+ 184us) server_negotiation.cc:407] Connection header received
0926 20:23:39.712851 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:39.712855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:39.712909 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:39.712995 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:23:39.713850 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.714376 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.715241 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.715472 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.715643 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:39.717057 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:23:39.717061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:23:39.717064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:23:39.717098 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:23:39.718467 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:39.718908 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:39.718914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:39.718916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:39.719001 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:23:39.719275 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:23:39.719279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:23:39.719281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:23:39.719613 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:23:39.719753 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:39.720023 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 20:23:39.720147 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:23:39.724014 15875 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:23:39.727154 15875 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:23:39.742348 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:39.736738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37492 (local address 127.0.116.195:38389)
0926 20:23:39.737204 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:39.737217 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:39.737241 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:23:39.737463 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:39.737470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:39.737692 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:39.737844 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:23:39.738473 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.739357 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.740781 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.741656 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.741761 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:39.741933 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:39.742082 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 20:23:39.742181 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":136,"threads_started":1}
I20260926 20:23:39.742336 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:39.736581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:38389 (local address 127.0.0.1:37492)
0926 20:23:39.737056 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:39.737117 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:39.737252 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:39.738066 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:39.738076 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:39.738101 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:23:39.738340 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:39.738351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.739529 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.739535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:39.740663 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:39.740671 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.741761 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.741764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:39.741872 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:39.741910 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 20:23:39.742052 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 20:23:39.742180 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":60,"thread_start_us":103,"threads_started":1}
I20260926 20:23:39.742574 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:39.736597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:43847 (local address 127.0.0.1:47314)
0926 20:23:39.737409 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:39.737423 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:39.737490 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:39.738233 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:39.738235 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:39.738249 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:23:39.738473 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:39.738479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.739566 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.739571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:39.740867 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:39.740878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.742172 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.742176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:39.742276 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:39.742297 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 20:23:39.742374 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:23:39.742441 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":115,"thread_start_us":95,"threads_started":1}
I20260926 20:23:39.742997 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:39.737475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47314 (local address 127.0.116.193:43847)
0926 20:23:39.737760 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:39.737777 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:39.737800 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:23:39.737909 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:39.737915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:39.738112 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:39.738257 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:23:39.738592 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.739390 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.741012 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.741881 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.741980 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:39.742112 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:39.742745 (+ 633us) server_negotiation.cc:299] Negotiation successful
0926 20:23:39.742860 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":125,"threads_started":1}
I20260926 20:23:39.744388 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:39.737371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:42889 (local address 127.0.0.1:48530)
0926 20:23:39.738019 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:39.738032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:39.738119 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:39.740168 (+ 2049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:39.740172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:39.740192 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:23:39.740282 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:39.740287 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.741649 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.741652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:39.742745 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:39.742755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.743998 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.744002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:39.744087 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:39.744103 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 20:23:39.744195 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 20:23:39.744261 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":90,"threads_started":1}
I20260926 20:23:39.744458 16352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:39.737787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48530 (local address 127.0.116.194:42889)
0926 20:23:39.739693 (+ 1906us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:39.739705 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:39.739730 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:23:39.739811 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:39.739817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:39.740007 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:39.740168 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:23:39.740404 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.741502 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.742889 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.743717 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.743816 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:39.743967 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:39.744227 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 20:23:39.744319 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1793,"thread_start_us":144,"threads_started":1}
I20260926 20:23:39.746140 16270 tablet_service.cc:1467] Processing CreateTablet for tablet 6c30f59cd0534079ba75c93e7a5bacfd (DEFAULT_TABLE table=test-table [id=6527b2bc580e4153a31c5cbdf2d5d533]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:23:39.746140 16001 tablet_service.cc:1467] Processing CreateTablet for tablet 6c30f59cd0534079ba75c93e7a5bacfd (DEFAULT_TABLE table=test-table [id=6527b2bc580e4153a31c5cbdf2d5d533]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:23:39.747339 16270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c30f59cd0534079ba75c93e7a5bacfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:23:39.747334 16001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c30f59cd0534079ba75c93e7a5bacfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:23:39.747031 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 6c30f59cd0534079ba75c93e7a5bacfd (DEFAULT_TABLE table=test-table [id=6527b2bc580e4153a31c5cbdf2d5d533]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:23:39.747964 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c30f59cd0534079ba75c93e7a5bacfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:23:39.754845 16353 tablet_bootstrap.cc:492] T 6c30f59cd0534079ba75c93e7a5bacfd P b90889308e1941e28b5eb97e45b1f913: Bootstrap starting.
I20260926 20:23:39.755800 16354 tablet_bootstrap.cc:492] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799: Bootstrap starting.
I20260926 20:23:39.755874 16355 tablet_bootstrap.cc:492] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe: Bootstrap starting.
I20260926 20:23:39.757543 16353 tablet_bootstrap.cc:654] T 6c30f59cd0534079ba75c93e7a5bacfd P b90889308e1941e28b5eb97e45b1f913: Neither blocks nor log segments found. Creating new log.
I20260926 20:23:39.758036 16354 tablet_bootstrap.cc:654] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799: Neither blocks nor log segments found. Creating new log.
I20260926 20:23:39.758036 16355 tablet_bootstrap.cc:654] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe: Neither blocks nor log segments found. Creating new log.
I20260926 20:23:39.758395 16353 log.cc:824] T 6c30f59cd0534079ba75c93e7a5bacfd P b90889308e1941e28b5eb97e45b1f913: Log is configured to *not* fsync() on all Append() calls
I20260926 20:23:39.758951 16355 log.cc:824] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe: Log is configured to *not* fsync() on all Append() calls
I20260926 20:23:39.759282 16354 log.cc:824] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799: Log is configured to *not* fsync() on all Append() calls
I20260926 20:23:39.760823 16353 tablet_bootstrap.cc:492] T 6c30f59cd0534079ba75c93e7a5bacfd P b90889308e1941e28b5eb97e45b1f913: No bootstrap required, opened a new log
I20260926 20:23:39.761044 16353 ts_tablet_manager.cc:1397] T 6c30f59cd0534079ba75c93e7a5bacfd P b90889308e1941e28b5eb97e45b1f913: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:23:39.761044 16355 tablet_bootstrap.cc:492] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe: No bootstrap required, opened a new log
I20260926 20:23:39.761242 16355 ts_tablet_manager.cc:1397] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:23:39.761270 16354 tablet_bootstrap.cc:492] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799: No bootstrap required, opened a new log
I20260926 20:23:39.761422 16354 ts_tablet_manager.cc:1397] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260926 20:23:39.764411 16353 raft_consensus.cc:348] T 6c30f59cd0534079ba75c93e7a5bacfd P b90889308e1941e28b5eb97e45b1f913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } }
I20260926 20:23:39.764671 16353 raft_consensus.cc:374] T 6c30f59cd0534079ba75c93e7a5bacfd P b90889308e1941e28b5eb97e45b1f913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:23:39.764744 16353 raft_consensus.cc:729] T 6c30f59cd0534079ba75c93e7a5bacfd P b90889308e1941e28b5eb97e45b1f913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90889308e1941e28b5eb97e45b1f913, State: Initialized, Role: FOLLOWER
I20260926 20:23:39.765260 16353 consensus_queue.cc:260] T 6c30f59cd0534079ba75c93e7a5bacfd P b90889308e1941e28b5eb97e45b1f913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } }
I20260926 20:23:39.765228 16354 raft_consensus.cc:348] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } }
I20260926 20:23:39.765530 16354 raft_consensus.cc:374] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:23:39.765645 16354 raft_consensus.cc:729] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc16bee212ed4616bbf66ff7dc175799, State: Initialized, Role: FOLLOWER
I20260926 20:23:39.765543 16355 raft_consensus.cc:348] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } }
I20260926 20:23:39.765826 16355 raft_consensus.cc:374] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:23:39.765914 16355 raft_consensus.cc:729] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77b5a0d96c94986a49ba05cc9a068fe, State: Initialized, Role: FOLLOWER
I20260926 20:23:39.765960 16201 heartbeater.cc:503] Master 127.0.116.254:45107 was elected leader, sending a full tablet report...
I20260926 20:23:39.766178 16354 consensus_queue.cc:260] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } }
I20260926 20:23:39.766330 16353 ts_tablet_manager.cc:1428] T 6c30f59cd0534079ba75c93e7a5bacfd P b90889308e1941e28b5eb97e45b1f913: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 20:23:39.766454 16355 consensus_queue.cc:260] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } }
I20260926 20:23:39.767246 16354 ts_tablet_manager.cc:1428] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260926 20:23:39.767412 16355 ts_tablet_manager.cc:1428] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 20:23:39.767647 16336 heartbeater.cc:503] Master 127.0.116.254:45107 was elected leader, sending a full tablet report...
I20260926 20:23:39.768692 16067 heartbeater.cc:503] Master 127.0.116.254:45107 was elected leader, sending a full tablet report...
W20260926 20:23:39.940179 16337 tablet.cc:2369] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:23:39.957857 16360 raft_consensus.cc:482] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:23:39.958081 16360 raft_consensus.cc:504] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } }
I20260926 20:23:39.959385 16360 leader_election.cc:288] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e77b5a0d96c94986a49ba05cc9a068fe (127.0.116.193:43847), b90889308e1941e28b5eb97e45b1f913 (127.0.116.194:42889)
W20260926 20:23:39.963779 16202 tablet.cc:2369] T 6c30f59cd0534079ba75c93e7a5bacfd P b90889308e1941e28b5eb97e45b1f913: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:23:39.964242 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:39.959693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:43847 (local address 127.0.116.195:42055)
0926 20:23:39.959818 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:39.959836 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:39.959956 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:39.960274 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:39.960277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:39.960299 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:23:39.960445 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:39.960454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.961482 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.961485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:39.962724 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:39.962734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.963781 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.963786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:39.963902 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:39.963928 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 20:23:39.964007 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 20:23:39.964091 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 20:23:39.964326 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:39.959843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:42055 (local address 127.0.116.193:43847)
0926 20:23:39.959997 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:39.960004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:39.960023 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:23:39.960088 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:39.960093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:39.960158 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:39.960240 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:23:39.960571 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.961355 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.962875 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.963569 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.963704 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:39.963762 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:39.964094 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 20:23:39.964186 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:23:39.964359 16352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:39.959951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:35553 (local address 127.0.116.194:42889)
0926 20:23:39.960104 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:39.960108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:39.960493 (+ 385us) server_negotiation.cc:407] Connection header received
0926 20:23:39.960611 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:39.960614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:39.960668 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:39.960743 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:23:39.961012 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.961634 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.963028 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.963677 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.963845 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:39.963890 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:39.964155 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 20:23:39.964251 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:23:39.964720 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:39.959693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:42889 (local address 127.0.116.195:35553)
0926 20:23:39.960389 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:39.960407 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:23:39.960506 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:23:39.960754 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:23:39.960756 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:23:39.960770 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:23:39.960907 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:39.960912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.961730 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.961733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:39.962914 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:23:39.962921 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:23:39.963852 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:23:39.963856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:23:39.963984 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:39.964006 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:23:39.964482 (+ 476us) client_negotiation.cc:240] Negotiation successful
0926 20:23:39.964577 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":309,"thread_start_us":155,"threads_started":1}
I20260926 20:23:39.965099 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c30f59cd0534079ba75c93e7a5bacfd" candidate_uuid: "bc16bee212ed4616bbf66ff7dc175799" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" is_pre_election: true
I20260926 20:23:39.965304 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c30f59cd0534079ba75c93e7a5bacfd" candidate_uuid: "bc16bee212ed4616bbf66ff7dc175799" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b90889308e1941e28b5eb97e45b1f913" is_pre_election: true
I20260926 20:23:39.965444 16021 raft_consensus.cc:2454] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc16bee212ed4616bbf66ff7dc175799 in term 0.
I20260926 20:23:39.965552 16155 raft_consensus.cc:2454] T 6c30f59cd0534079ba75c93e7a5bacfd P b90889308e1941e28b5eb97e45b1f913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc16bee212ed4616bbf66ff7dc175799 in term 0.
I20260926 20:23:39.965999 16222 leader_election.cc:302] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b90889308e1941e28b5eb97e45b1f913, bc16bee212ed4616bbf66ff7dc175799, e77b5a0d96c94986a49ba05cc9a068fe; no voters:
I20260926 20:23:39.966322 16360 raft_consensus.cc:2781] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:23:39.966403 16360 raft_consensus.cc:482] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:23:39.966432 16360 raft_consensus.cc:3037] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:23:39.967573 16360 raft_consensus.cc:504] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } }
I20260926 20:23:39.967935 16360 leader_election.cc:288] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799 [CANDIDATE]: Term 1 election: Requested vote from peers e77b5a0d96c94986a49ba05cc9a068fe (127.0.116.193:43847), b90889308e1941e28b5eb97e45b1f913 (127.0.116.194:42889)
I20260926 20:23:39.968345 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c30f59cd0534079ba75c93e7a5bacfd" candidate_uuid: "bc16bee212ed4616bbf66ff7dc175799" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77b5a0d96c94986a49ba05cc9a068fe"
I20260926 20:23:39.968349 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c30f59cd0534079ba75c93e7a5bacfd" candidate_uuid: "bc16bee212ed4616bbf66ff7dc175799" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b90889308e1941e28b5eb97e45b1f913"
I20260926 20:23:39.968456 16021 raft_consensus.cc:3037] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:23:39.968487 16155 raft_consensus.cc:3037] T 6c30f59cd0534079ba75c93e7a5bacfd P b90889308e1941e28b5eb97e45b1f913 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:23:39.969475 16155 raft_consensus.cc:2454] T 6c30f59cd0534079ba75c93e7a5bacfd P b90889308e1941e28b5eb97e45b1f913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc16bee212ed4616bbf66ff7dc175799 in term 1.
I20260926 20:23:39.969529 16021 raft_consensus.cc:2454] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc16bee212ed4616bbf66ff7dc175799 in term 1.
I20260926 20:23:39.969828 16222 leader_election.cc:302] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b90889308e1941e28b5eb97e45b1f913, bc16bee212ed4616bbf66ff7dc175799; no voters:
I20260926 20:23:39.970007 16360 raft_consensus.cc:2781] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:23:39.970206 16360 raft_consensus.cc:686] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799 [term 1 LEADER]: Becoming Leader. State: Replica: bc16bee212ed4616bbf66ff7dc175799, State: Running, Role: LEADER
I20260926 20:23:39.970567 16360 consensus_queue.cc:237] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } }
I20260926 20:23:39.973727 15873 catalog_manager.cc:5495] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc16bee212ed4616bbf66ff7dc175799 (127.0.116.195). New cstate: current_term: 1 leader_uuid: "bc16bee212ed4616bbf66ff7dc175799" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } health_report { overall_health: UNKNOWN } } }
W20260926 20:23:39.982683 16068 tablet.cc:2369] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:23:40.039235 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:40.035625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.0.116.195:38389)
0926 20:23:40.035787 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 20:23:40.035792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:23:40.035808 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:23:40.035934 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:23:40.035937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:23:40.035992 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:23:40.036110 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:23:40.036346 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:40.036862 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:40.037750 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:23:40.037969 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:23:40.038077 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:23:40.038386 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:23:40.038816 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:23:40.038956 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 20:23:40.038999 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 20:23:40.039057 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:23:40.045292 16155 raft_consensus.cc:1261] T 6c30f59cd0534079ba75c93e7a5bacfd P b90889308e1941e28b5eb97e45b1f913 [term 1 FOLLOWER]: Refusing update from remote peer bc16bee212ed4616bbf66ff7dc175799: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:23:40.045343 16021 raft_consensus.cc:1261] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe [term 1 FOLLOWER]: Refusing update from remote peer bc16bee212ed4616bbf66ff7dc175799: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:23:40.046042 16360 consensus_queue.cc:1035] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:23:40.046288 16363 consensus_queue.cc:1035] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:23:40.054625 16368 mvcc.cc:205] Tried to move back new op lower bound from 7333700485297905664 to 7333700485001965568. Current Snapshot: MvccSnapshot[applied={T|T < 7333700485297905664}]
I20260926 20:23:40.064461 15875 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:23:40.064633 15875 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:23:40.067471 15875 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400: Sending DeleteTablet for 3 replicas of tablet 6c30f59cd0534079ba75c93e7a5bacfd
I20260926 20:23:40.068533 16001 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c30f59cd0534079ba75c93e7a5bacfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:23:40 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:47314
I20260926 20:23:40.068535 16135 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c30f59cd0534079ba75c93e7a5bacfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:23:40 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:48530
I20260926 20:23:40.068631 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 15938
I20260926 20:23:40.068739 16270 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c30f59cd0534079ba75c93e7a5bacfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:23:40 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:37492
I20260926 20:23:40.069497 16377 tablet_replica.cc:330] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799: stopping tablet replica
I20260926 20:23:40.069741 16376 tablet_replica.cc:330] T 6c30f59cd0534079ba75c93e7a5bacfd P b90889308e1941e28b5eb97e45b1f913: stopping tablet replica
I20260926 20:23:40.069867 16377 raft_consensus.cc:2229] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:23:40.069988 16376 raft_consensus.cc:2229] T 6c30f59cd0534079ba75c93e7a5bacfd P b90889308e1941e28b5eb97e45b1f913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:23:40.070283 16377 raft_consensus.cc:2258] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:23:40.070322 16376 raft_consensus.cc:2258] T 6c30f59cd0534079ba75c93e7a5bacfd P b90889308e1941e28b5eb97e45b1f913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:23:40.073621 16376 ts_tablet_manager.cc:1905] T 6c30f59cd0534079ba75c93e7a5bacfd P b90889308e1941e28b5eb97e45b1f913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:23:40.073714 16377 ts_tablet_manager.cc:1905] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:23:40.076653 16377 ts_tablet_manager.cc:1918] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:23:40.076819 16377 log.cc:1196] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/6c30f59cd0534079ba75c93e7a5bacfd
I20260926 20:23:40.077186 16377 ts_tablet_manager.cc:1939] T 6c30f59cd0534079ba75c93e7a5bacfd P bc16bee212ed4616bbf66ff7dc175799: Deleting consensus metadata
I20260926 20:23:40.077236 16376 ts_tablet_manager.cc:1918] T 6c30f59cd0534079ba75c93e7a5bacfd P b90889308e1941e28b5eb97e45b1f913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:23:40.077394 16376 log.cc:1196] T 6c30f59cd0534079ba75c93e7a5bacfd P b90889308e1941e28b5eb97e45b1f913: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/6c30f59cd0534079ba75c93e7a5bacfd
I20260926 20:23:40.077759 16376 ts_tablet_manager.cc:1939] T 6c30f59cd0534079ba75c93e7a5bacfd P b90889308e1941e28b5eb97e45b1f913: Deleting consensus metadata
W20260926 20:23:40.078085 15863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.116.193:43847 (error 108)
I20260926 20:23:40.078557 15860 catalog_manager.cc:4841] TS bc16bee212ed4616bbf66ff7dc175799 (127.0.116.195:38389): tablet 6c30f59cd0534079ba75c93e7a5bacfd (table test-table [id=6527b2bc580e4153a31c5cbdf2d5d533]) successfully deleted
I20260926 20:23:40.079025 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 16072
I20260926 20:23:40.079113 15860 catalog_manager.cc:4841] TS b90889308e1941e28b5eb97e45b1f913 (127.0.116.194:42889): tablet 6c30f59cd0534079ba75c93e7a5bacfd (table test-table [id=6527b2bc580e4153a31c5cbdf2d5d533]) successfully deleted
I20260926 20:23:40.079697 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:23:40.079338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:43847 (local address 127.0.0.1:47318)
0926 20:23:40.079481 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 20:23:40.079593 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.193:43847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 20:23:40.079861 15863 catalog_manager.cc:4584] TS e77b5a0d96c94986a49ba05cc9a068fe (127.0.116.193:43847): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c30f59cd0534079ba75c93e7a5bacfd: Network error: Client connection negotiation failed: client connection to 127.0.116.193:43847: connect: Connection refused (error 111)
I20260926 20:23:40.085970 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 16206
I20260926 20:23:40.091679 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 15844
I20260926 20:24:00.098313 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:45107
--webserver_interface=127.0.116.254
--webserver_port=43421
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:35573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:45107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:24:00.214241 16379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:00.214541 16379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:00.214612 16379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:00.218739 16379 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 20:24:00.218801 16379 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 20:24:00.218817 16379 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 20:24:00.218832 16379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:24:00.218850 16379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:00.218900 16379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:00.218933 16379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:24:00.218953 16379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:24:00.224540 16379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:35573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:45107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:45107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=43421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:00.225646 16379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:00.226799 16379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:00.240777 16386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:00.240782 16388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:00.240800 16385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:00.241010 16379 server_base.cc:1034] running on GCE node
I20260926 20:24:00.241950 16379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:00.243387 16379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:00.244585 16379 hybrid_clock.cc:648] HybridClock initialized: now 1790454240244551 us; error 46 us; skew 500 ppm
Sep 26 20:24:00 dist-test-slave-qsm9 krb5kdc[15831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:00.248027 16379 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:24:00.249112 16379 webserver.cc:460] Webserver started at http://127.0.116.254:43421/ using document root <none> and password file <none>
I20260926 20:24:00.249768 16379 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:00.249879 16379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:00.254138 16379 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:24:00.256573 16395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:00.258009 16379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:24:00.258155 16379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "33516122d3ae4d59a744fd889e0b7400"
format_stamp: "Formatted at 2026-09-26 20:23:38 on dist-test-slave-qsm9"
server_key: "8079862cee6a536c39c9f0db86e727fe"
server_key_iv: "201831c6e49559e059fbc052f1f05882"
server_key_version: "encryptionkey@0"
I20260926 20:24:00.258787 16379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:00.274006 16379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:00.277691 16379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:00.277910 16379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:00.285629 16379 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:45107
I20260926 20:24:00.285939 16447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:45107 every 8 connection(s)
I20260926 20:24:00.286911 16379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:24:00.291637 16448 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:24:00.293471 16448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400: Bootstrap starting.
I20260926 20:24:00.295871 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 16379
I20260926 20:24:00.296381 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:43847
--local_ip_for_outbound_sockets=127.0.116.193
--tserver_master_addrs=127.0.116.254:45107
--webserver_port=45871
--webserver_interface=127.0.116.193
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:35573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:24:00.312163 16448 log.cc:824] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:00.321633 16448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:24:00.322065 16448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400: Bootstrap complete.
I20260926 20:24:00.326373 16448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33516122d3ae4d59a744fd889e0b7400" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45107 } }
I20260926 20:24:00.326853 16448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33516122d3ae4d59a744fd889e0b7400, State: Initialized, Role: FOLLOWER
I20260926 20:24:00.327469 16448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33516122d3ae4d59a744fd889e0b7400" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45107 } }
I20260926 20:24:00.327683 16448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:24:00.327786 16448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:24:00.327909 16448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:24:00.330037 16448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33516122d3ae4d59a744fd889e0b7400" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45107 } }
I20260926 20:24:00.330504 16448 leader_election.cc:302] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33516122d3ae4d59a744fd889e0b7400; no voters:
I20260926 20:24:00.331010 16448 leader_election.cc:288] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:24:00.331120 16452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:24:00.331404 16452 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 2 LEADER]: Becoming Leader. State: Replica: 33516122d3ae4d59a744fd889e0b7400, State: Running, Role: LEADER
I20260926 20:24:00.331841 16452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33516122d3ae4d59a744fd889e0b7400" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45107 } }
I20260926 20:24:00.332384 16448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:24:00.333407 16453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "33516122d3ae4d59a744fd889e0b7400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33516122d3ae4d59a744fd889e0b7400" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45107 } } }
I20260926 20:24:00.333537 16453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:00.334331 16454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33516122d3ae4d59a744fd889e0b7400. Latest consensus state: current_term: 2 leader_uuid: "33516122d3ae4d59a744fd889e0b7400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33516122d3ae4d59a744fd889e0b7400" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45107 } } }
I20260926 20:24:00.334442 16454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:00.337452 16468 catalog_manager.cc:1259] Loaded cluster ID: 22541c5ff5604266b729cd3f8379f1a0
I20260926 20:24:00.337532 16468 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400: loading cluster ID for follower catalog manager: success
I20260926 20:24:00.339309 16468 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400: acquiring CA information for follower catalog manager: success
I20260926 20:24:00.340600 16468 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:24:00.340745 16461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:24:00.342796 16461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:24:00.343242 16461 catalog_manager.cc:1259] Loaded cluster ID: 22541c5ff5604266b729cd3f8379f1a0
I20260926 20:24:00.343317 16461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:24:00.343955 16461 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:24:00.344363 16461 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400: Loaded TSK: 0
I20260926 20:24:00.364544 16461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400: Generated new TSK 1
I20260926 20:24:00.364862 16461 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:24:00.430632 16450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:00.430958 16450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:00.431042 16450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:00.435618 16450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:00.435703 16450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:00.435817 16450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:24:00.441308 16450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:35573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:43847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=45871
--tserver_master_addrs=127.0.116.254:45107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:00.442440 16450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:00.443598 16450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:00.454066 16478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:00.454057 16476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:00.454058 16475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:00.454579 16450 server_base.cc:1034] running on GCE node
I20260926 20:24:00.455179 16450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:00.455964 16450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:00.457187 16450 hybrid_clock.cc:648] HybridClock initialized: now 1790454240457158 us; error 50 us; skew 500 ppm
Sep 26 20:24:00 dist-test-slave-qsm9 krb5kdc[15831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:00.460289 16450 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:24:00.461316 16450 webserver.cc:460] Webserver started at http://127.0.116.193:45871/ using document root <none> and password file <none>
I20260926 20:24:00.462007 16450 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:00.462090 16450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:00.466773 16450 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:24:00.469435 16485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:00.470947 16450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:24:00.471109 16450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "e77b5a0d96c94986a49ba05cc9a068fe"
format_stamp: "Formatted at 2026-09-26 20:23:39 on dist-test-slave-qsm9"
server_key: "405981b7f7342f311bee0fc3ca5d9d7b"
server_key_iv: "1d847ab139783c77d5e7332ca1e1447f"
server_key_version: "encryptionkey@0"
I20260926 20:24:00.471786 16450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:00.484619 16450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:00.488202 16450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:00.488412 16450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:00.489076 16450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:00.490347 16492 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:24:00.496054 16450 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:24:00.496176 16450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 20:24:00.496281 16450 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:24:00.498215 16450 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:24:00.498306 16450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 20:24:00.498769 16492 tablet_bootstrap.cc:492] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe: Bootstrap starting.
I20260926 20:24:00.509481 16450 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:43847
I20260926 20:24:00.509557 16599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:43847 every 8 connection(s)
I20260926 20:24:00.510699 16450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:24:00.514284 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 16450
I20260926 20:24:00.514945 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:42889
--local_ip_for_outbound_sockets=127.0.116.194
--tserver_master_addrs=127.0.116.254:45107
--webserver_port=39795
--webserver_interface=127.0.116.194
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:35573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:24:00.520366 16492 log.cc:824] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:24:00 dist-test-slave-qsm9 krb5kdc[15831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:24:00.530040 16602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:00.512949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.193:50123)
0926 20:24:00.513500 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:00.513560 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:00.514549 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:00.516024 (+ 1475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:00.516032 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:00.516562 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:00.517326 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:00.517350 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.519449 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.519452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:00.520618 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:00.520630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.521790 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.521794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:00.521993 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:00.522680 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:24:00.522697 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:00.524355 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:00.526643 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:00.526649 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:00.526664 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:00.526947 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:00.527320 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:00.527324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:00.527326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:00.527480 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:00.528143 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:00.528149 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:00.528420 (+ 271us) client_negotiation.cc:769] Sending connection context
0926 20:24:00.528628 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 20:24:00.528993 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":155,"threads_started":1}
I20260926 20:24:00.530862 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:00.512989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:50123 (local address 127.0.116.254:45107)
0926 20:24:00.513879 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:00.513901 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:00.513954 (+ 53us) server_negotiation.cc:407] Connection header received
0926 20:24:00.515026 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:00.515039 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:00.515564 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:00.515948 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:00.517570 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.519260 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.520802 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.521623 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.521829 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:00.524596 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:00.524602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:00.524615 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:00.524647 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:00.526506 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:00.527077 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:00.527088 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:00.527091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:00.527214 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:00.527615 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:00.527619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:00.527621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:00.527996 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:00.528176 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:00.528784 (+ 608us) server_negotiation.cc:299] Negotiation successful
0926 20:24:00.529063 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":296,"threads_started":1}
I20260926 20:24:00.532094 16492 tablet_bootstrap.cc:492] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:24:00.532608 16492 tablet_bootstrap.cc:492] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe: Bootstrap complete.
I20260926 20:24:00.533397 16600 heartbeater.cc:347] Connected to a master server at 127.0.116.254:45107
I20260926 20:24:00.533658 16600 heartbeater.cc:464] Registering TS with master...
I20260926 20:24:00.534157 16492 ts_tablet_manager.cc:1397] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe: Time spent bootstrapping tablet: real 0.036s user 0.030s sys 0.004s
I20260926 20:24:00.534279 16600 heartbeater.cc:511] Master 127.0.116.254:45107 requested a full tablet report, sending...
I20260926 20:24:00.536979 16412 ts_manager.cc:194] Registered new tserver with Master: e77b5a0d96c94986a49ba05cc9a068fe (127.0.116.193:43847)
I20260926 20:24:00.537401 16492 raft_consensus.cc:348] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } }
I20260926 20:24:00.538014 16492 raft_consensus.cc:729] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77b5a0d96c94986a49ba05cc9a068fe, State: Initialized, Role: FOLLOWER
I20260926 20:24:00.538561 16492 consensus_queue.cc:260] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } }
I20260926 20:24:00.539637 16492 ts_tablet_manager.cc:1428] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 20:24:00.540932 16412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:50123
I20260926 20:24:00.543705 16600 heartbeater.cc:503] Master 127.0.116.254:45107 was elected leader, sending a full tablet report...
Sep 26 20:24:00 dist-test-slave-qsm9 krb5kdc[15831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for kudu/127.0.116.193@KRBTEST.COM
I20260926 20:24:00.552131 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:00.540371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:43847 (local address 127.0.0.1:50140)
0926 20:24:00.541228 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:00.541275 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:00.541433 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:00.542065 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:00.542073 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:00.542550 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:00.542699 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:00.542708 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.544061 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.544064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:00.545059 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:00.545071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.545857 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.545860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:00.545951 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:00.546539 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:24:00.546550 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:00.548021 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:00.550205 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:00.550213 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:00.550218 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:00.550470 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:00.550786 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:00.550788 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:00.550789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:00.550877 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:00.551381 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:00.551385 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:00.551570 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 20:24:00.551767 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 20:24:00.551996 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":118,"threads_started":1}
I20260926 20:24:00.552211 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:00.540448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50140 (local address 127.0.116.193:43847)
0926 20:24:00.540811 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:00.540823 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:00.541383 (+ 560us) server_negotiation.cc:407] Connection header received
0926 20:24:00.541670 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:00.541675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:00.541895 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:00.542043 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:00.542844 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.543916 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.545208 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.545724 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.545835 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:00.548168 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:00.548178 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:00.548191 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:00.548218 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:00.550085 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:00.550587 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:00.550592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:00.550594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:00.550691 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:00.550966 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:00.550969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:00.550971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:00.551266 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:00.551398 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:00.551866 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 20:24:00.552043 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":271,"thread_start_us":97,"threads_started":1}
I20260926 20:24:00.553450 16534 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c30f59cd0534079ba75c93e7a5bacfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:23:40 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:50140
I20260926 20:24:00.554386 16609 tablet_replica.cc:330] stopping tablet replica
I20260926 20:24:00.554698 16609 raft_consensus.cc:2229] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:24:00.554963 16609 raft_consensus.cc:2258] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:24:00.556242 16609 ts_tablet_manager.cc:1905] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:24:00.560367 16609 ts_tablet_manager.cc:1918] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:24:00.560472 16609 log.cc:1196] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/wal/wals/6c30f59cd0534079ba75c93e7a5bacfd
I20260926 20:24:00.560787 16609 ts_tablet_manager.cc:1939] T 6c30f59cd0534079ba75c93e7a5bacfd P e77b5a0d96c94986a49ba05cc9a068fe: Deleting consensus metadata
I20260926 20:24:00.563647 16399 catalog_manager.cc:4841] TS e77b5a0d96c94986a49ba05cc9a068fe (127.0.116.193:43847): tablet 6c30f59cd0534079ba75c93e7a5bacfd (table test-table [id=6527b2bc580e4153a31c5cbdf2d5d533]) successfully deleted
W20260926 20:24:00.644311 16604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:00.644623 16604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:00.644716 16604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:00.648914 16604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:00.649022 16604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:00.649116 16604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:24:00.654989 16604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:35573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:42889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=39795
--tserver_master_addrs=127.0.116.254:45107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:00.656256 16604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:00.657351 16604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:00.667886 16614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:00.667886 16617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:00.668130 16604 server_base.cc:1034] running on GCE node
W20260926 20:24:00.667954 16615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:00.668773 16604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:00.669520 16604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:00.670706 16604 hybrid_clock.cc:648] HybridClock initialized: now 1790454240670683 us; error 34 us; skew 500 ppm
Sep 26 20:24:00 dist-test-slave-qsm9 krb5kdc[15831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:00.673969 16604 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:24:00.675091 16604 webserver.cc:460] Webserver started at http://127.0.116.194:39795/ using document root <none> and password file <none>
I20260926 20:24:00.675853 16604 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:00.675964 16604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:00.680320 16604 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:24:00.682806 16624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:00.684342 16604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:24:00.684491 16604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "b90889308e1941e28b5eb97e45b1f913"
format_stamp: "Formatted at 2026-09-26 20:23:39 on dist-test-slave-qsm9"
server_key: "3b410352a9c77998c7068e763a51a835"
server_key_iv: "40839f8e2c7c061971a06872c8ce749d"
server_key_version: "encryptionkey@0"
I20260926 20:24:00.685209 16604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:00.705356 16604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:00.708675 16604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:00.708901 16604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:00.709538 16604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:00.710417 16604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:24:00.710496 16604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:00.710572 16604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:24:00.710616 16604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:00.720815 16604 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:42889
I20260926 20:24:00.720844 16737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:42889 every 8 connection(s)
I20260926 20:24:00.721897 16604 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:24:00.724933 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 16604
I20260926 20:24:00.725447 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:38389
--local_ip_for_outbound_sockets=127.0.116.195
--tserver_master_addrs=127.0.116.254:45107
--webserver_port=34049
--webserver_interface=127.0.116.195
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:35573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:24:00 dist-test-slave-qsm9 krb5kdc[15831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:24:00.738317 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:00.723923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:35855 (local address 127.0.116.254:45107)
0926 20:24:00.724092 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:00.724097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:00.725145 (+ 1048us) server_negotiation.cc:407] Connection header received
0926 20:24:00.727174 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:00.727180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:00.727260 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:00.727376 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:00.728766 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.729327 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.730597 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.730864 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.730984 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:00.734374 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:00.734378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:00.734381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:00.734420 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:00.736113 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:00.736681 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:00.736684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:00.736686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:00.736794 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:00.737154 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:00.737159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:00.737161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:00.737368 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:00.737474 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:00.737996 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 20:24:00.738135 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:24:00.739203 16740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:00.724406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.194:35855)
0926 20:24:00.724937 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:00.725000 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:00.726869 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:00.727628 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:00.727636 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:00.728096 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:00.728601 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:00.728614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.729460 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.729466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:00.730465 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:00.730475 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.731030 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.731034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:00.731217 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:00.732122 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:24:00.732140 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:00.734170 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:00.736248 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:00.736256 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:00.736274 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:00.736555 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:00.736890 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:00.736895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:00.736897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:00.737055 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:00.737473 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:00.737481 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:00.737758 (+ 277us) client_negotiation.cc:769] Sending connection context
0926 20:24:00.737957 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 20:24:00.738308 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":114,"threads_started":1}
I20260926 20:24:00.740571 16738 heartbeater.cc:347] Connected to a master server at 127.0.116.254:45107
I20260926 20:24:00.740891 16738 heartbeater.cc:464] Registering TS with master...
I20260926 20:24:00.741434 16738 heartbeater.cc:511] Master 127.0.116.254:45107 requested a full tablet report, sending...
I20260926 20:24:00.742583 16412 ts_manager.cc:194] Registered new tserver with Master: b90889308e1941e28b5eb97e45b1f913 (127.0.116.194:42889)
I20260926 20:24:00.743358 16412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:35855
W20260926 20:24:00.841306 16741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:00.841620 16741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:00.841665 16741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:00.845805 16741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:00.845875 16741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:00.845965 16741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:24:00.851552 16741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:35573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:38389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=34049
--tserver_master_addrs=127.0.116.254:45107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:00.852656 16741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:00.853713 16741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:00.863755 16747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:00.863752 16746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:00.864125 16741 server_base.cc:1034] running on GCE node
W20260926 20:24:00.863744 16749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:00.864691 16741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:00.865324 16741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:00.866529 16741 hybrid_clock.cc:648] HybridClock initialized: now 1790454240866509 us; error 52 us; skew 500 ppm
Sep 26 20:24:00 dist-test-slave-qsm9 krb5kdc[15831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:00.869480 16741 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:24:00.870342 16741 webserver.cc:460] Webserver started at http://127.0.116.195:34049/ using document root <none> and password file <none>
I20260926 20:24:00.870911 16741 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:00.871037 16741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:00.875669 16741 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:24:00.878095 16756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:00.879755 16741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:24:00.879942 16741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "bc16bee212ed4616bbf66ff7dc175799"
format_stamp: "Formatted at 2026-09-26 20:23:39 on dist-test-slave-qsm9"
server_key: "69b2813d14c632061e625ad8e4c78a8c"
server_key_iv: "dc992dca010579bfc27c679a75f94d49"
server_key_version: "encryptionkey@0"
I20260926 20:24:00.880578 16741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:00.901525 16741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:00.905555 16741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:00.905790 16741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:00.906462 16741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:00.907379 16741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:24:00.907470 16741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:00.907600 16741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:24:00.907649 16741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:00.917724 16741 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:38389
I20260926 20:24:00.917750 16869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:38389 every 8 connection(s)
I20260926 20:24:00.919041 16741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:24:00.923810 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 16741
Sep 26 20:24:00 dist-test-slave-qsm9 krb5kdc[15831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:24:00.932855 16873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:00.924573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37044 (local address 127.0.116.254:45107)
0926 20:24:00.924882 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:00.924887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:00.925009 (+ 122us) server_negotiation.cc:407] Connection header received
0926 20:24:00.925190 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:00.925192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:00.925236 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:00.925329 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:00.926192 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.926709 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.927636 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.927919 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.928033 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:00.929477 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:00.929484 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:00.929488 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:00.929517 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:00.931072 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:00.931530 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:00.931535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:00.931537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:00.931620 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:00.931902 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:00.931906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:00.931907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:00.932095 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:00.932361 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:00.932492 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 20:24:00.932662 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":232,"thread_start_us":96,"threads_started":1}
I20260926 20:24:00.934927 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:00.921370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:36941 (local address 127.0.116.254:45107)
0926 20:24:00.921557 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:00.921562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:00.922548 (+ 986us) server_negotiation.cc:407] Connection header received
0926 20:24:00.923951 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:00.923955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:00.924017 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:00.924118 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:00.925577 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.926356 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.927613 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.927895 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.928004 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:00.930863 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:00.930872 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:00.930874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:00.930897 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:00.932360 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:00.933049 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:00.933052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:00.933054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:00.933131 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:00.933553 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:00.933556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:00.933557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:00.933803 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:00.933895 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:00.934630 (+ 735us) server_negotiation.cc:299] Negotiation successful
0926 20:24:00.934729 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":73,"spinlock_wait_cycles":36224}
I20260926 20:24:00.936010 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:00.921838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.195:36941)
0926 20:24:00.922354 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:00.922399 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:00.923709 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:00.924333 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:00.924346 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:00.924902 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:00.925437 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:00.925452 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.926480 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.926484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:00.927444 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:00.927469 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.928052 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.928056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:00.928258 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:00.928900 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:24:00.928916 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:00.930691 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:00.932480 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:00.932487 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:00.932503 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:00.932883 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:00.933222 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:00.933226 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:00.933228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:00.933458 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:00.933987 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:00.933994 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:00.934364 (+ 370us) client_negotiation.cc:769] Sending connection context
0926 20:24:00.934588 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 20:24:00.934976 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":136,"threads_started":1}
I20260926 20:24:00.937263 16870 heartbeater.cc:347] Connected to a master server at 127.0.116.254:45107
I20260926 20:24:00.937598 16870 heartbeater.cc:464] Registering TS with master...
I20260926 20:24:00.938161 16870 heartbeater.cc:511] Master 127.0.116.254:45107 requested a full tablet report, sending...
I20260926 20:24:00.939209 16412 ts_manager.cc:194] Registered new tserver with Master: bc16bee212ed4616bbf66ff7dc175799 (127.0.116.195:38389)
I20260926 20:24:00.939836 16412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:36941
I20260926 20:24:00.946983 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:24:00.950968 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:00.947635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37052 (local address 127.0.116.254:45107)
0926 20:24:00.947756 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:00.947759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:00.948023 (+ 264us) server_negotiation.cc:407] Connection header received
0926 20:24:00.948164 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:00.948167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:00.948267 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:00.948356 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:24:00.948570 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.949068 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.949905 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.950113 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.950189 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:00.950535 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:24:00.950671 (+ 136us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 20:24:00.950865 (+ 194us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37052: token expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 20:24:00.951052 16603 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37052: token expired
W20260926 20:24:00.951442 16339 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.116.254:45107)
I20260926 20:24:00.959044 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:00.951894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37064 (local address 127.0.116.254:45107)
0926 20:24:00.952048 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:00.952054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:00.952069 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:24:00.952205 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:00.952208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:00.952253 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:00.952334 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:00.952951 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.953425 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.954216 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.954432 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.954539 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:00.956010 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:00.956014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:00.956016 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:00.956047 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:00.957391 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:00.957819 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:00.957823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:00.957825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:00.957910 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:00.958193 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:00.958197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:00.958198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:00.958381 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:00.958523 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:00.958754 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 20:24:00.958870 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 20:24:00.962033 16412 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:24:00.965911 16412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:24:00.973953 16534 tablet_service.cc:1467] Processing CreateTablet for tablet 32362f05d9534bb2a797c837170d4f85 (DEFAULT_TABLE table=test-table [id=c0601caa3cb949b3b7a3e495056f47b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:24:00.974669 16534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32362f05d9534bb2a797c837170d4f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:00.977679 16492 tablet_bootstrap.cc:492] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe: Bootstrap starting.
I20260926 20:24:00.978578 16492 tablet_bootstrap.cc:654] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:00.978924 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:00.972904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:38389 (local address 127.0.0.1:54618)
0926 20:24:00.973433 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:00.973449 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:00.973542 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:00.974165 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:00.974169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:00.974187 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:00.974297 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:00.974304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.975964 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.975968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:00.977199 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:00.977210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.978357 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.978360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:00.978478 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:00.978552 (+ 74us) client_negotiation.cc:769] Sending connection context
0926 20:24:00.978645 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 20:24:00.978744 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":81,"spinlock_wait_cycles":1792}
I20260926 20:24:00.978965 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:00.973093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54618 (local address 127.0.116.195:38389)
0926 20:24:00.973632 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:00.973645 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:00.973669 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:24:00.973750 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:00.973755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:00.973985 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:00.974477 (+ 492us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:00.974639 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.975835 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.977344 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.978222 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.978337 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:00.978490 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:00.978707 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 20:24:00.978820 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":174,"threads_started":1}
I20260926 20:24:00.979686 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:00.973138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:42889 (local address 127.0.0.1:33614)
0926 20:24:00.973433 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:00.973449 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:00.973542 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:00.974473 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:00.974476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:00.974494 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:00.974593 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:00.974599 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.975667 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.975671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:00.977199 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:00.977210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.979221 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.979227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:00.979316 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:00.979341 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:24:00.979423 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 20:24:00.979541 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":98,"spinlock_wait_cycles":6784,"thread_start_us":106,"threads_started":1}
I20260926 20:24:00.980158 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:00.973713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33614 (local address 127.0.116.194:42889)
0926 20:24:00.974011 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:00.974024 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:00.974048 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:24:00.974125 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:00.974130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:00.974333 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:00.974501 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:00.974711 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.975547 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.977344 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:00.978222 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:00.978337 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:00.978490 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:00.979796 (+ 1306us) server_negotiation.cc:299] Negotiation successful
0926 20:24:00.980009 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":113,"threads_started":1}
I20260926 20:24:00.981884 16804 tablet_service.cc:1467] Processing CreateTablet for tablet 32362f05d9534bb2a797c837170d4f85 (DEFAULT_TABLE table=test-table [id=c0601caa3cb949b3b7a3e495056f47b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:24:00.982924 16804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32362f05d9534bb2a797c837170d4f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:00.983081 16492 tablet_bootstrap.cc:492] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe: No bootstrap required, opened a new log
I20260926 20:24:00.983168 16492 ts_tablet_manager.cc:1397] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260926 20:24:00.983660 16492 raft_consensus.cc:348] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } }
I20260926 20:24:00.983785 16492 raft_consensus.cc:374] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:00.983824 16492 raft_consensus.cc:729] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77b5a0d96c94986a49ba05cc9a068fe, State: Initialized, Role: FOLLOWER
I20260926 20:24:00.983958 16492 consensus_queue.cc:260] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } }
I20260926 20:24:00.984345 16492 ts_tablet_manager.cc:1428] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 20:24:00.985378 16672 tablet_service.cc:1467] Processing CreateTablet for tablet 32362f05d9534bb2a797c837170d4f85 (DEFAULT_TABLE table=test-table [id=c0601caa3cb949b3b7a3e495056f47b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:24:00.986399 16672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32362f05d9534bb2a797c837170d4f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:00.991015 16883 tablet_bootstrap.cc:492] T 32362f05d9534bb2a797c837170d4f85 P bc16bee212ed4616bbf66ff7dc175799: Bootstrap starting.
I20260926 20:24:00.993628 16884 tablet_bootstrap.cc:492] T 32362f05d9534bb2a797c837170d4f85 P b90889308e1941e28b5eb97e45b1f913: Bootstrap starting.
I20260926 20:24:00.993780 16883 tablet_bootstrap.cc:654] T 32362f05d9534bb2a797c837170d4f85 P bc16bee212ed4616bbf66ff7dc175799: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:00.994671 16883 log.cc:824] T 32362f05d9534bb2a797c837170d4f85 P bc16bee212ed4616bbf66ff7dc175799: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:00.996018 16884 tablet_bootstrap.cc:654] T 32362f05d9534bb2a797c837170d4f85 P b90889308e1941e28b5eb97e45b1f913: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:00.996608 16883 tablet_bootstrap.cc:492] T 32362f05d9534bb2a797c837170d4f85 P bc16bee212ed4616bbf66ff7dc175799: No bootstrap required, opened a new log
I20260926 20:24:00.996758 16884 log.cc:824] T 32362f05d9534bb2a797c837170d4f85 P b90889308e1941e28b5eb97e45b1f913: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:00.996811 16883 ts_tablet_manager.cc:1397] T 32362f05d9534bb2a797c837170d4f85 P bc16bee212ed4616bbf66ff7dc175799: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:24:00.998835 16884 tablet_bootstrap.cc:492] T 32362f05d9534bb2a797c837170d4f85 P b90889308e1941e28b5eb97e45b1f913: No bootstrap required, opened a new log
I20260926 20:24:00.999047 16884 ts_tablet_manager.cc:1397] T 32362f05d9534bb2a797c837170d4f85 P b90889308e1941e28b5eb97e45b1f913: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:24:01.000407 16883 raft_consensus.cc:348] T 32362f05d9534bb2a797c837170d4f85 P bc16bee212ed4616bbf66ff7dc175799 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } }
I20260926 20:24:01.000677 16883 raft_consensus.cc:374] T 32362f05d9534bb2a797c837170d4f85 P bc16bee212ed4616bbf66ff7dc175799 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:01.000759 16883 raft_consensus.cc:729] T 32362f05d9534bb2a797c837170d4f85 P bc16bee212ed4616bbf66ff7dc175799 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc16bee212ed4616bbf66ff7dc175799, State: Initialized, Role: FOLLOWER
I20260926 20:24:01.001252 16883 consensus_queue.cc:260] T 32362f05d9534bb2a797c837170d4f85 P bc16bee212ed4616bbf66ff7dc175799 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } }
I20260926 20:24:01.001992 16870 heartbeater.cc:503] Master 127.0.116.254:45107 was elected leader, sending a full tablet report...
I20260926 20:24:01.002292 16883 ts_tablet_manager.cc:1428] T 32362f05d9534bb2a797c837170d4f85 P bc16bee212ed4616bbf66ff7dc175799: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260926 20:24:01.002240 16884 raft_consensus.cc:348] T 32362f05d9534bb2a797c837170d4f85 P b90889308e1941e28b5eb97e45b1f913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } }
I20260926 20:24:01.002524 16884 raft_consensus.cc:374] T 32362f05d9534bb2a797c837170d4f85 P b90889308e1941e28b5eb97e45b1f913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:01.002580 16884 raft_consensus.cc:729] T 32362f05d9534bb2a797c837170d4f85 P b90889308e1941e28b5eb97e45b1f913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90889308e1941e28b5eb97e45b1f913, State: Initialized, Role: FOLLOWER
I20260926 20:24:01.003001 16884 consensus_queue.cc:260] T 32362f05d9534bb2a797c837170d4f85 P b90889308e1941e28b5eb97e45b1f913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } }
I20260926 20:24:01.003702 16738 heartbeater.cc:503] Master 127.0.116.254:45107 was elected leader, sending a full tablet report...
I20260926 20:24:01.005218 16884 ts_tablet_manager.cc:1428] T 32362f05d9534bb2a797c837170d4f85 P b90889308e1941e28b5eb97e45b1f913: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260926 20:24:01.012123 16601 tablet.cc:2369] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:24:01.102728 16606 raft_consensus.cc:482] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:24:01.103013 16606 raft_consensus.cc:504] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } }
I20260926 20:24:01.104228 16606 leader_election.cc:288] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc16bee212ed4616bbf66ff7dc175799 (127.0.116.195:38389), b90889308e1941e28b5eb97e45b1f913 (127.0.116.194:42889)
I20260926 20:24:01.108632 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:01.104722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:58837 (local address 127.0.116.195:38389)
0926 20:24:01.104864 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:01.104868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:01.105093 (+ 225us) server_negotiation.cc:407] Connection header received
0926 20:24:01.105348 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:01.105352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:01.105400 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:01.105445 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:01.105829 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:01.106394 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:01.107553 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:01.108004 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:01.108145 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:01.108204 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:01.108418 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 20:24:01.108490 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:24:01.108637 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:01.104636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:38389 (local address 127.0.116.193:58837)
0926 20:24:01.104992 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:01.105011 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:01.105142 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:01.105575 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:01.105578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:01.105595 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:01.105698 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:01.105702 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:01.106520 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:01.106527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:01.107392 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:01.107401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:01.108134 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:01.108139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:01.108252 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:01.108300 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 20:24:01.108401 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 20:24:01.108482 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":100,"threads_started":1}
I20260926 20:24:01.109131 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:01.105232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:40209 (local address 127.0.116.194:42889)
0926 20:24:01.105357 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:01.105363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:01.105602 (+ 239us) server_negotiation.cc:407] Connection header received
0926 20:24:01.105838 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:01.105842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:01.105902 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:01.105996 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:01.106287 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:01.106877 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:01.108059 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:01.108518 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:01.108661 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:01.108723 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:01.108914 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 20:24:01.109004 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:24:01.109513 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:01.105086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:42889 (local address 127.0.116.193:40209)
0926 20:24:01.105357 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:01.105371 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:01.105684 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:01.106007 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:01.106010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:01.106027 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:01.106136 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:01.106142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:01.107003 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:01.107009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:01.107933 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:01.107942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:01.108648 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:01.108652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:01.108752 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:01.108780 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 20:24:01.109335 (+ 555us) client_negotiation.cc:240] Negotiation successful
0926 20:24:01.109412 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":71,"threads_started":1}
I20260926 20:24:01.109526 16824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32362f05d9534bb2a797c837170d4f85" candidate_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc16bee212ed4616bbf66ff7dc175799" is_pre_election: true
I20260926 20:24:01.109925 16824 raft_consensus.cc:2454] T 32362f05d9534bb2a797c837170d4f85 P bc16bee212ed4616bbf66ff7dc175799 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e77b5a0d96c94986a49ba05cc9a068fe in term 0.
I20260926 20:24:01.110338 16692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32362f05d9534bb2a797c837170d4f85" candidate_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b90889308e1941e28b5eb97e45b1f913" is_pre_election: true
I20260926 20:24:01.110489 16486 leader_election.cc:302] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc16bee212ed4616bbf66ff7dc175799, e77b5a0d96c94986a49ba05cc9a068fe; no voters:
I20260926 20:24:01.110651 16692 raft_consensus.cc:2454] T 32362f05d9534bb2a797c837170d4f85 P b90889308e1941e28b5eb97e45b1f913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e77b5a0d96c94986a49ba05cc9a068fe in term 0.
I20260926 20:24:01.110834 16606 raft_consensus.cc:2781] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:24:01.110934 16606 raft_consensus.cc:482] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:24:01.111003 16606 raft_consensus.cc:3037] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:01.112502 16606 raft_consensus.cc:504] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } }
I20260926 20:24:01.112982 16606 leader_election.cc:288] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [CANDIDATE]: Term 1 election: Requested vote from peers bc16bee212ed4616bbf66ff7dc175799 (127.0.116.195:38389), b90889308e1941e28b5eb97e45b1f913 (127.0.116.194:42889)
I20260926 20:24:01.113461 16824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32362f05d9534bb2a797c837170d4f85" candidate_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc16bee212ed4616bbf66ff7dc175799"
I20260926 20:24:01.113554 16692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32362f05d9534bb2a797c837170d4f85" candidate_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b90889308e1941e28b5eb97e45b1f913"
I20260926 20:24:01.113598 16824 raft_consensus.cc:3037] T 32362f05d9534bb2a797c837170d4f85 P bc16bee212ed4616bbf66ff7dc175799 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:01.113731 16692 raft_consensus.cc:3037] T 32362f05d9534bb2a797c837170d4f85 P b90889308e1941e28b5eb97e45b1f913 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:01.114809 16692 raft_consensus.cc:2454] T 32362f05d9534bb2a797c837170d4f85 P b90889308e1941e28b5eb97e45b1f913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b5a0d96c94986a49ba05cc9a068fe in term 1.
I20260926 20:24:01.114845 16824 raft_consensus.cc:2454] T 32362f05d9534bb2a797c837170d4f85 P bc16bee212ed4616bbf66ff7dc175799 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e77b5a0d96c94986a49ba05cc9a068fe in term 1.
I20260926 20:24:01.115157 16486 leader_election.cc:302] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc16bee212ed4616bbf66ff7dc175799, e77b5a0d96c94986a49ba05cc9a068fe; no voters:
I20260926 20:24:01.115342 16606 raft_consensus.cc:2781] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:24:01.115585 16606 raft_consensus.cc:686] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 1 LEADER]: Becoming Leader. State: Replica: e77b5a0d96c94986a49ba05cc9a068fe, State: Running, Role: LEADER
I20260926 20:24:01.115880 16606 consensus_queue.cc:237] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } }
I20260926 20:24:01.118988 16412 catalog_manager.cc:5495] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe reported cstate change: term changed from 0 to 1, leader changed from <none> to e77b5a0d96c94986a49ba05cc9a068fe (127.0.116.193). New cstate: current_term: 1 leader_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } health_report { overall_health: HEALTHY } } }
I20260926 20:24:01.138427 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:01.134332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50148 (local address 127.0.116.193:43847)
0926 20:24:01.134690 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:01.134747 (+ 57us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:01.134764 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:24:01.134852 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:01.134854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:01.134918 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:01.135044 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:24:01.135261 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:01.135992 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:01.136818 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:01.137102 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:01.137209 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:01.137489 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:24:01.137994 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:24:01.138101 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:01.138224 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 20:24:01.138284 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":188,"threads_started":1}
I20260926 20:24:01.143028 16692 raft_consensus.cc:1261] T 32362f05d9534bb2a797c837170d4f85 P b90889308e1941e28b5eb97e45b1f913 [term 1 FOLLOWER]: Refusing update from remote peer e77b5a0d96c94986a49ba05cc9a068fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:24:01.143028 16824 raft_consensus.cc:1261] T 32362f05d9534bb2a797c837170d4f85 P bc16bee212ed4616bbf66ff7dc175799 [term 1 FOLLOWER]: Refusing update from remote peer e77b5a0d96c94986a49ba05cc9a068fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:24:01.143889 16891 consensus_queue.cc:1035] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:24:01.144084 16606 consensus_queue.cc:1035] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:24:01.153589 16895 mvcc.cc:205] Tried to move back new op lower bound from 7333700571713986560 to 7333700571613167616. Current Snapshot: MvccSnapshot[applied={T|T < 7333700571713986560 or (T in {7333700571713986560})}]
I20260926 20:24:01.153857 16897 mvcc.cc:205] Tried to move back new op lower bound from 7333700571713986560 to 7333700571613167616. Current Snapshot: MvccSnapshot[applied={T|T < 7333700571713986560}]
I20260926 20:24:01.154039 16896 mvcc.cc:205] Tried to move back new op lower bound from 7333700571713986560 to 7333700571613167616. Current Snapshot: MvccSnapshot[applied={T|T < 7333700571713986560}]
I20260926 20:24:01.161252 16412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:24:01.161473 16412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:24:01.164024 16412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400: Sending DeleteTablet for 3 replicas of tablet 32362f05d9534bb2a797c837170d4f85
I20260926 20:24:01.164798 16804 tablet_service.cc:1514] Processing DeleteTablet for tablet 32362f05d9534bb2a797c837170d4f85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:24:01 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:54618
I20260926 20:24:01.164855 16672 tablet_service.cc:1514] Processing DeleteTablet for tablet 32362f05d9534bb2a797c837170d4f85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:24:01 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:33614
I20260926 20:24:01.165132 16534 tablet_service.cc:1514] Processing DeleteTablet for tablet 32362f05d9534bb2a797c837170d4f85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:24:01 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:50140
I20260926 20:24:01.165535 16904 tablet_replica.cc:330] T 32362f05d9534bb2a797c837170d4f85 P bc16bee212ed4616bbf66ff7dc175799: stopping tablet replica
I20260926 20:24:01.165563 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 16450
I20260926 20:24:01.165925 16904 raft_consensus.cc:2229] T 32362f05d9534bb2a797c837170d4f85 P bc16bee212ed4616bbf66ff7dc175799 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:24:01.166320 16904 raft_consensus.cc:2258] T 32362f05d9534bb2a797c837170d4f85 P bc16bee212ed4616bbf66ff7dc175799 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:24:01.166334 16905 tablet_replica.cc:330] T 32362f05d9534bb2a797c837170d4f85 P b90889308e1941e28b5eb97e45b1f913: stopping tablet replica
I20260926 20:24:01.166649 16905 raft_consensus.cc:2229] T 32362f05d9534bb2a797c837170d4f85 P b90889308e1941e28b5eb97e45b1f913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:24:01.166950 16905 raft_consensus.cc:2258] T 32362f05d9534bb2a797c837170d4f85 P b90889308e1941e28b5eb97e45b1f913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:24:01.168666 16905 ts_tablet_manager.cc:1905] T 32362f05d9534bb2a797c837170d4f85 P b90889308e1941e28b5eb97e45b1f913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:24:01.169027 16904 ts_tablet_manager.cc:1905] T 32362f05d9534bb2a797c837170d4f85 P bc16bee212ed4616bbf66ff7dc175799: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:24:01.171411 16905 ts_tablet_manager.cc:1918] T 32362f05d9534bb2a797c837170d4f85 P b90889308e1941e28b5eb97e45b1f913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:24:01.171574 16905 log.cc:1196] T 32362f05d9534bb2a797c837170d4f85 P b90889308e1941e28b5eb97e45b1f913: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/32362f05d9534bb2a797c837170d4f85
I20260926 20:24:01.172014 16905 ts_tablet_manager.cc:1939] T 32362f05d9534bb2a797c837170d4f85 P b90889308e1941e28b5eb97e45b1f913: Deleting consensus metadata
I20260926 20:24:01.172824 16904 ts_tablet_manager.cc:1918] T 32362f05d9534bb2a797c837170d4f85 P bc16bee212ed4616bbf66ff7dc175799: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:24:01.172927 16904 log.cc:1196] T 32362f05d9534bb2a797c837170d4f85 P bc16bee212ed4616bbf66ff7dc175799: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/32362f05d9534bb2a797c837170d4f85
I20260926 20:24:01.173300 16904 ts_tablet_manager.cc:1939] T 32362f05d9534bb2a797c837170d4f85 P bc16bee212ed4616bbf66ff7dc175799: Deleting consensus metadata
I20260926 20:24:01.173380 16396 catalog_manager.cc:4841] TS b90889308e1941e28b5eb97e45b1f913 (127.0.116.194:42889): tablet 32362f05d9534bb2a797c837170d4f85 (table test-table [id=c0601caa3cb949b3b7a3e495056f47b1]) successfully deleted
W20260926 20:24:01.173955 16399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.116.193:43847 (error 108)
I20260926 20:24:01.174182 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 16604
I20260926 20:24:01.175022 16396 catalog_manager.cc:4841] TS bc16bee212ed4616bbf66ff7dc175799 (127.0.116.195:38389): tablet 32362f05d9534bb2a797c837170d4f85 (table test-table [id=c0601caa3cb949b3b7a3e495056f47b1]) successfully deleted
I20260926 20:24:01.176019 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:01.175720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:43847 (local address 127.0.0.1:50160)
0926 20:24:01.175868 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:01.175945 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.193:43847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 20:24:01.176231 16399 catalog_manager.cc:4584] TS e77b5a0d96c94986a49ba05cc9a068fe (127.0.116.193:43847): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 32362f05d9534bb2a797c837170d4f85: Network error: Client connection negotiation failed: client connection to 127.0.116.193:43847: connect: Connection refused (error 111)
I20260926 20:24:01.180683 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 16741
I20260926 20:24:01.186811 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 16379
I20260926 20:24:21.193902 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:45107
--webserver_interface=127.0.116.254
--webserver_port=43421
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:35573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:45107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:24:21.309314 16908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:21.309600 16908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:21.309643 16908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:21.313905 16908 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 20:24:21.313970 16908 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 20:24:21.313987 16908 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 20:24:21.314000 16908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:24:21.314020 16908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:21.314038 16908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:21.314054 16908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:24:21.314069 16908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:24:21.319449 16908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:35573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:45107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:45107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=43421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:21.320657 16908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:21.321728 16908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:24:21.331173 16908 server_base.cc:1034] running on GCE node
W20260926 20:24:21.331120 16913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:21.331121 16916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:21.331120 16914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:21.332043 16908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:21.333214 16908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:21.334419 16908 hybrid_clock.cc:648] HybridClock initialized: now 1790454261334397 us; error 34 us; skew 500 ppm
Sep 26 20:24:21 dist-test-slave-qsm9 krb5kdc[15831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:21.337432 16908 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:24:21.338387 16908 webserver.cc:460] Webserver started at http://127.0.116.254:43421/ using document root <none> and password file <none>
I20260926 20:24:21.338976 16908 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:21.339040 16908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:21.343394 16908 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:24:21.346221 16923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:21.347669 16908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:24:21.347781 16908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "33516122d3ae4d59a744fd889e0b7400"
format_stamp: "Formatted at 2026-09-26 20:23:38 on dist-test-slave-qsm9"
server_key: "8079862cee6a536c39c9f0db86e727fe"
server_key_iv: "201831c6e49559e059fbc052f1f05882"
server_key_version: "encryptionkey@0"
I20260926 20:24:21.348366 16908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:21.373301 16908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:21.376927 16908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:21.377094 16908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:21.384490 16908 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:45107
I20260926 20:24:21.384805 16975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:45107 every 8 connection(s)
I20260926 20:24:21.385671 16908 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:24:21.390223 16976 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:24:21.391781 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 16908
I20260926 20:24:21.392051 16976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400: Bootstrap starting.
I20260926 20:24:21.392268 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:43847
--local_ip_for_outbound_sockets=127.0.116.193
--tserver_master_addrs=127.0.116.254:45107
--webserver_port=45871
--webserver_interface=127.0.116.193
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:35573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:24:21.411746 16976 log.cc:824] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:21.422987 16976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:24:21.423396 16976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400: Bootstrap complete.
I20260926 20:24:21.427377 16976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33516122d3ae4d59a744fd889e0b7400" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45107 } }
I20260926 20:24:21.427846 16976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33516122d3ae4d59a744fd889e0b7400, State: Initialized, Role: FOLLOWER
I20260926 20:24:21.428350 16976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33516122d3ae4d59a744fd889e0b7400" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45107 } }
I20260926 20:24:21.428463 16976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:24:21.428556 16976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:24:21.428673 16976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 2 FOLLOWER]: Advancing to term 3
I20260926 20:24:21.430588 16976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33516122d3ae4d59a744fd889e0b7400" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45107 } }
I20260926 20:24:21.430943 16976 leader_election.cc:302] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33516122d3ae4d59a744fd889e0b7400; no voters:
I20260926 20:24:21.431290 16976 leader_election.cc:288] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 20:24:21.431392 16980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 20:24:21.431609 16980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [term 3 LEADER]: Becoming Leader. State: Replica: 33516122d3ae4d59a744fd889e0b7400, State: Running, Role: LEADER
I20260926 20:24:21.431975 16980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33516122d3ae4d59a744fd889e0b7400" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45107 } }
I20260926 20:24:21.432597 16976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:24:21.432971 16980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33516122d3ae4d59a744fd889e0b7400. Latest consensus state: current_term: 3 leader_uuid: "33516122d3ae4d59a744fd889e0b7400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33516122d3ae4d59a744fd889e0b7400" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45107 } } }
I20260926 20:24:21.433084 16980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:21.433483 16981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "33516122d3ae4d59a744fd889e0b7400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33516122d3ae4d59a744fd889e0b7400" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45107 } } }
I20260926 20:24:21.433657 16981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400 [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:21.434719 16984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:24:21.440076 16984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:24:21.440694 16984 catalog_manager.cc:1259] Loaded cluster ID: 22541c5ff5604266b729cd3f8379f1a0
I20260926 20:24:21.440757 16984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:24:21.443205 16984 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:24:21.444252 16984 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400: Loaded TSK: 0
I20260926 20:24:21.444366 16984 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 33516122d3ae4d59a744fd889e0b7400: Loaded TSK: 1
F20260926 20:24:21.445201 16984 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4f2db4b9ad google::LogMessage::Fail() at ??:0
@ 0x7f4f2db4e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f4f2db4b4de google::LogMessage::Flush() at ??:0
@ 0x7f4f2db4ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4f3080f18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f4f30807f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f4f384b47bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f4f3848e99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f4f3848f178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f4f384d0df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f4f384ef82b std::function<>::operator()() at ??:0
@ 0x7f4f3848eb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f4f3848ff3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4f3848c579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f4f384d0640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4f2fd9d3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4f2fd9dc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4f2fd9f3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4f2fd8e512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4f313d26db start_thread at ??:0
@ 0x7f4f2c64961f clone at ??:0
W20260926 20:24:21.540771 16978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:21.541074 16978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:21.541121 16978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:21.545790 16978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:21.545848 16978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:21.545922 16978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:24:21.552314 16978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:35573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:43847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=45871
--tserver_master_addrs=127.0.116.254:45107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:21.553437 16978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:21.554410 16978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:21.565403 17023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:21.565357 17026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:21.565357 17024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:21.566012 16978 server_base.cc:1034] running on GCE node
I20260926 20:24:21.566509 16978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:21.567323 16978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:21.568540 16978 hybrid_clock.cc:648] HybridClock initialized: now 1790454261568504 us; error 63 us; skew 500 ppm
Sep 26 20:24:21 dist-test-slave-qsm9 krb5kdc[15831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:21.571767 16978 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:24:21.572885 16978 webserver.cc:460] Webserver started at http://127.0.116.193:45871/ using document root <none> and password file <none>
I20260926 20:24:21.573490 16978 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:21.573559 16978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:21.577888 16978 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260926 20:24:21.580500 17033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:21.581923 16978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:24:21.582073 16978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "e77b5a0d96c94986a49ba05cc9a068fe"
format_stamp: "Formatted at 2026-09-26 20:23:39 on dist-test-slave-qsm9"
server_key: "405981b7f7342f311bee0fc3ca5d9d7b"
server_key_iv: "1d847ab139783c77d5e7332ca1e1447f"
server_key_version: "encryptionkey@0"
I20260926 20:24:21.582649 16978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:21.610190 16978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:21.614030 16978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:21.614269 16978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:21.614975 16978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:21.616264 17040 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:24:21.621932 16978 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:24:21.622018 16978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 20:24:21.622112 16978 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:24:21.623929 16978 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:24:21.624006 16978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260926 20:24:21.624311 17040 tablet_bootstrap.cc:492] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe: Bootstrap starting.
I20260926 20:24:21.634889 16978 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:43847
I20260926 20:24:21.635070 17148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:43847 every 8 connection(s)
I20260926 20:24:21.636023 16978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:24:21.642401 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 16978
I20260926 20:24:21.642967 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:42889
--local_ip_for_outbound_sockets=127.0.116.194
--tserver_master_addrs=127.0.116.254:45107
--webserver_port=39795
--webserver_interface=127.0.116.194
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:35573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:24:21.645184 17040 log.cc:824] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:21.655874 17040 tablet_bootstrap.cc:492] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:24:21.656324 17040 tablet_bootstrap.cc:492] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe: Bootstrap complete.
I20260926 20:24:21.657729 17040 ts_tablet_manager.cc:1397] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe: Time spent bootstrapping tablet: real 0.034s user 0.031s sys 0.000s
I20260926 20:24:21.661131 17040 raft_consensus.cc:348] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } }
I20260926 20:24:21.661682 17040 raft_consensus.cc:729] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77b5a0d96c94986a49ba05cc9a068fe, State: Initialized, Role: FOLLOWER
I20260926 20:24:21.662233 17040 consensus_queue.cc:260] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } }
I20260926 20:24:21.663475 17040 ts_tablet_manager.cc:1428] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260926 20:24:21.693389 17151 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:24:21.638409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.193:42369)
0926 20:24:21.639039 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:21.639102 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:21.640196 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:21.692297 (+ 52101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":173,"threads_started":1}
W20260926 20:24:21.694164 17149 heartbeater.cc:644] Failed to heartbeat to 127.0.116.254:45107 (0 consecutive failures): Network error: Failed to ping master at 127.0.116.254:45107: Client connection negotiation failed: client connection to 127.0.116.254:45107: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:24:21.694985 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:21.694677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.193:37477)
0926 20:24:21.694827 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:21.694905 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 20:24:21.695796 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:21.695568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.193:55377)
0926 20:24:21.695682 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:21.695734 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 20:24:21.696062 17149 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:24:21.765506 17152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:21.765846 17152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:21.765929 17152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:21.771668 17152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:21.771761 17152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:21.771868 17152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:24:21.777547 17152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:35573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:42889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=39795
--tserver_master_addrs=127.0.116.254:45107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:21.778760 17152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:21.779923 17152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:21.789784 17162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:21.789804 17159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:21.789899 17152 server_base.cc:1034] running on GCE node
W20260926 20:24:21.790102 17160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:21.790553 17152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:21.791215 17152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:21.792397 17152 hybrid_clock.cc:648] HybridClock initialized: now 1790454261792375 us; error 78 us; skew 500 ppm
Sep 26 20:24:21 dist-test-slave-qsm9 krb5kdc[15831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:21.795209 17152 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:24:21.796171 17152 webserver.cc:460] Webserver started at http://127.0.116.194:39795/ using document root <none> and password file <none>
I20260926 20:24:21.796759 17152 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:21.796854 17152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:21.800889 17152 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:24:21.803068 17169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:21.804723 17152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:24:21.804873 17152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "b90889308e1941e28b5eb97e45b1f913"
format_stamp: "Formatted at 2026-09-26 20:23:39 on dist-test-slave-qsm9"
server_key: "3b410352a9c77998c7068e763a51a835"
server_key_iv: "40839f8e2c7c061971a06872c8ce749d"
server_key_version: "encryptionkey@0"
I20260926 20:24:21.805464 17152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:21.824635 17152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:21.828019 17152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:21.828233 17152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:21.828840 17152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:21.829660 17152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:24:21.829730 17152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:21.829802 17152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:24:21.829854 17152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:21.839695 17152 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:42889
I20260926 20:24:21.839712 17282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:42889 every 8 connection(s)
I20260926 20:24:21.840715 17152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:24:21.842496 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 17152
I20260926 20:24:21.843012 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:38389
--local_ip_for_outbound_sockets=127.0.116.195
--tserver_master_addrs=127.0.116.254:45107
--webserver_port=34049
--webserver_interface=127.0.116.195
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:35573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:24:21.845058 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:21.843006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.194:59697)
0926 20:24:21.843527 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:21.843658 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":139,"threads_started":1}
W20260926 20:24:21.845642 17283 heartbeater.cc:644] Failed to heartbeat to 127.0.116.254:45107 (0 consecutive failures): Network error: Failed to ping master at 127.0.116.254:45107: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
I20260926 20:24:21.846331 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:21.846100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.194:48089)
0926 20:24:21.846218 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:21.846268 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 20:24:21.846956 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:21.846753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.194:56739)
0926 20:24:21.846851 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:21.846896 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 20:24:21.847203 17283 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:24:21.963197 17286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:21.963583 17286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:21.963701 17286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:21.968001 17286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:21.968084 17286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:21.968185 17286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:24:21.973922 17286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:35573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:38389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=34049
--tserver_master_addrs=127.0.116.254:45107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:21.975127 17286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:21.976356 17286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:21.986987 17292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:21.987077 17294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:21.987092 17291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:21.987536 17286 server_base.cc:1034] running on GCE node
I20260926 20:24:21.987982 17286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:21.988704 17286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:21.989928 17286 hybrid_clock.cc:648] HybridClock initialized: now 1790454261989888 us; error 51 us; skew 500 ppm
Sep 26 20:24:21 dist-test-slave-qsm9 krb5kdc[15831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:21.993208 17286 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:24:21.994211 17286 webserver.cc:460] Webserver started at http://127.0.116.195:34049/ using document root <none> and password file <none>
I20260926 20:24:21.994776 17286 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:21.994848 17286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:21.999037 17286 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:24:22.001711 17301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:22.003046 17286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:24:22.003145 17286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "bc16bee212ed4616bbf66ff7dc175799"
format_stamp: "Formatted at 2026-09-26 20:23:39 on dist-test-slave-qsm9"
server_key: "69b2813d14c632061e625ad8e4c78a8c"
server_key_iv: "dc992dca010579bfc27c679a75f94d49"
server_key_version: "encryptionkey@0"
I20260926 20:24:22.003933 17286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:22.014178 17286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:22.017671 17286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:22.017836 17286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:22.018399 17286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:22.019223 17286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:24:22.019272 17286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:22.019315 17286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:24:22.019330 17286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:22.028862 17286 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:38389
I20260926 20:24:22.028899 17414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:38389 every 8 connection(s)
I20260926 20:24:22.029901 17286 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:24:22.030608 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 17286
I20260926 20:24:22.034080 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:22.032293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.195:51961)
0926 20:24:22.032740 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:22.032845 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":120,"threads_started":1}
W20260926 20:24:22.034580 17415 heartbeater.cc:644] Failed to heartbeat to 127.0.116.254:45107 (0 consecutive failures): Network error: Failed to ping master at 127.0.116.254:45107: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
I20260926 20:24:22.035185 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:22.034970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.195:51333)
0926 20:24:22.035080 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:22.035129 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 20:24:22.036015 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:22.035778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.195:38231)
0926 20:24:22.035907 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:22.035958 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 20:24:22.036254 17415 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:24:22.697191 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:22.696700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.193:51751)
0926 20:24:22.697000 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:22.697070 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":82,"threads_started":1}
I20260926 20:24:22.848218 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:22.847789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.194:47403)
0926 20:24:22.848047 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:22.848108 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":79,"threads_started":1}
I20260926 20:24:23.037321 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:23.036879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.195:56691)
0926 20:24:23.037143 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:23.037211 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
I20260926 20:24:23.521960 17422 raft_consensus.cc:482] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:24:23.522192 17422 raft_consensus.cc:504] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } }
I20260926 20:24:23.523249 17422 leader_election.cc:288] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc16bee212ed4616bbf66ff7dc175799 (127.0.116.195:38389), b90889308e1941e28b5eb97e45b1f913 (127.0.116.194:42889)
Sep 26 20:24:23 dist-test-slave-qsm9 krb5kdc[15831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.194@KRBTEST.COM
Sep 26 20:24:23 dist-test-slave-qsm9 krb5kdc[15831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.195@KRBTEST.COM
I20260926 20:24:23.543023 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:23.523975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:42889 (local address 127.0.116.193:54575)
0926 20:24:23.524211 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:23.524225 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:23.524332 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:23.525712 (+ 1380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:23.525715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:23.526269 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:23.526718 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:23.526733 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:23.528111 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:23.528115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:23.529065 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:23.529079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:23.529789 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:23.529793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:23.530055 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:23.535485 (+ 5430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:24:23.535576 (+ 91us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:23.537391 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:23.540318 (+ 2927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:23.540327 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:23.540343 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:23.540802 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:23.541246 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:23.541249 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:23.541250 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:23.541402 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:23.542182 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:23.542187 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:23.542412 (+ 225us) client_negotiation.cc:769] Sending connection context
0926 20:24:23.542567 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 20:24:23.542852 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":168,"spinlock_wait_cycles":50688,"thread_start_us":76,"threads_started":1}
I20260926 20:24:23.543346 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:23.523775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:35783 (local address 127.0.116.195:38389)
0926 20:24:23.524229 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:23.524251 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:23.524289 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:24:23.524981 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:23.524987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:23.525334 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:23.525656 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:23.526891 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:23.527963 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:23.529205 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:23.529666 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:23.529844 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:23.538025 (+ 8181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:23.538031 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:23.538044 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:23.538074 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:23.540350 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:23.540939 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:23.540952 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:23.540961 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:23.541099 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:23.541507 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:23.541513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:23.541518 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:23.542011 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:23.542181 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:23.542769 (+ 588us) server_negotiation.cc:299] Negotiation successful
0926 20:24:23.543137 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":135,"threads_started":1}
I20260926 20:24:23.543555 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:23.523575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:38389 (local address 127.0.116.193:35783)
0926 20:24:23.523820 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:23.523838 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:23.523941 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:23.525706 (+ 1765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:23.525712 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:23.526253 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:23.526718 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:23.526733 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:23.528124 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:23.528127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:23.529067 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:23.529081 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:23.529790 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:23.529794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:23.530030 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:23.535577 (+ 5547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:24:23.535591 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:23.537789 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:23.540492 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:23.540495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:23.540496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:23.540798 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:23.541207 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:23.541213 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:23.541216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:23.541395 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:23.542152 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:23.542160 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:23.542412 (+ 252us) client_negotiation.cc:769] Sending connection context
0926 20:24:23.542607 (+ 195us) client_negotiation.cc:240] Negotiation successful
0926 20:24:23.542958 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":284,"spinlock_wait_cycles":44800,"thread_start_us":77,"threads_started":1}
I20260926 20:24:23.544308 17426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:23.524163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:54575 (local address 127.0.116.194:42889)
0926 20:24:23.524567 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:23.524578 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:23.524608 (+ 30us) server_negotiation.cc:407] Connection header received
0926 20:24:23.525210 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:23.525214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:23.525497 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:23.525771 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:23.526900 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:23.527973 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:23.529200 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:23.529666 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:23.529852 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:23.537736 (+ 7884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:23.537746 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:23.537766 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:23.537805 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:23.540143 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:23.540979 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:23.540984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:23.540989 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:23.541114 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:23.541536 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:23.541538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:23.541540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:23.542035 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:23.543695 (+ 1660us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:23.543870 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 20:24:23.544160 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":86,"threads_started":1}
W20260926 20:24:23.546615 17034 leader_election.cc:341] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b90889308e1941e28b5eb97e45b1f913 (127.0.116.194:42889): Not found: Tablet not found: 32362f05d9534bb2a797c837170d4f85
W20260926 20:24:23.546821 17034 leader_election.cc:341] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc16bee212ed4616bbf66ff7dc175799 (127.0.116.195:38389): Not found: Tablet not found: 32362f05d9534bb2a797c837170d4f85
I20260926 20:24:23.546877 17034 leader_election.cc:302] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e77b5a0d96c94986a49ba05cc9a068fe; no voters: b90889308e1941e28b5eb97e45b1f913, bc16bee212ed4616bbf66ff7dc175799
I20260926 20:24:23.547111 17422 raft_consensus.cc:2726] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:24:23.698583 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:23.698186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.193:60181)
0926 20:24:23.698379 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:23.698462 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 20:24:23.849751 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:23.849289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.194:53167)
0926 20:24:23.849562 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:23.849630 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260926 20:24:24.038571 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:24.038123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.195:56687)
0926 20:24:24.038405 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:24.038467 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":74,"threads_started":1}
I20260926 20:24:24.699882 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:24.699332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.193:50645)
0926 20:24:24.699705 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:24.699770 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":75,"threads_started":1}
I20260926 20:24:24.851395 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:24.850756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.194:55031)
0926 20:24:24.851218 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:24.851282 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":101,"threads_started":1}
I20260926 20:24:25.040028 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:25.039547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.195:41867)
0926 20:24:25.039836 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:25.039900 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":70,"threads_started":1}
I20260926 20:24:25.376331 17432 raft_consensus.cc:482] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:24:25.376468 17432 raft_consensus.cc:504] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } }
I20260926 20:24:25.376827 17432 leader_election.cc:288] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc16bee212ed4616bbf66ff7dc175799 (127.0.116.195:38389), b90889308e1941e28b5eb97e45b1f913 (127.0.116.194:42889)
W20260926 20:24:25.377769 17034 leader_election.cc:341] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc16bee212ed4616bbf66ff7dc175799 (127.0.116.195:38389): Not found: Tablet not found: 32362f05d9534bb2a797c837170d4f85
W20260926 20:24:25.377897 17034 leader_election.cc:341] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b90889308e1941e28b5eb97e45b1f913 (127.0.116.194:42889): Not found: Tablet not found: 32362f05d9534bb2a797c837170d4f85
I20260926 20:24:25.377952 17034 leader_election.cc:302] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e77b5a0d96c94986a49ba05cc9a068fe; no voters: b90889308e1941e28b5eb97e45b1f913, bc16bee212ed4616bbf66ff7dc175799
I20260926 20:24:25.378142 17432 raft_consensus.cc:2726] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:24:25.701278 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:25.700743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.193:38577)
0926 20:24:25.701090 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:25.701162 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":110,"threads_started":1}
I20260926 20:24:25.852614 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:25.852175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.194:34451)
0926 20:24:25.852466 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:25.852522 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":136,"threads_started":1}
I20260926 20:24:26.041293 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:26.040799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.195:46181)
0926 20:24:26.041122 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:26.041183 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":90,"threads_started":1}
I20260926 20:24:26.703053 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:26.702427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.193:49539)
0926 20:24:26.702770 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:26.702932 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":99,"threads_started":1}
I20260926 20:24:26.854104 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:26.853592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.194:35093)
0926 20:24:26.853939 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:26.854007 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":104,"threads_started":1}
I20260926 20:24:27.042654 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:27.042166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.195:51825)
0926 20:24:27.042476 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:27.042555 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260926 20:24:27.081688 17439 raft_consensus.cc:482] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:24:27.082196 17439 raft_consensus.cc:504] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } }
I20260926 20:24:27.082824 17439 leader_election.cc:288] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc16bee212ed4616bbf66ff7dc175799 (127.0.116.195:38389), b90889308e1941e28b5eb97e45b1f913 (127.0.116.194:42889)
W20260926 20:24:27.083671 17034 leader_election.cc:341] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc16bee212ed4616bbf66ff7dc175799 (127.0.116.195:38389): Not found: Tablet not found: 32362f05d9534bb2a797c837170d4f85
W20260926 20:24:27.083854 17034 leader_election.cc:341] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b90889308e1941e28b5eb97e45b1f913 (127.0.116.194:42889): Not found: Tablet not found: 32362f05d9534bb2a797c837170d4f85
I20260926 20:24:27.083916 17034 leader_election.cc:302] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e77b5a0d96c94986a49ba05cc9a068fe; no voters: b90889308e1941e28b5eb97e45b1f913, bc16bee212ed4616bbf66ff7dc175799
I20260926 20:24:27.084082 17439 raft_consensus.cc:2726] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:24:27.704571 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:27.704094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.193:33063)
0926 20:24:27.704399 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:27.704467 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":80,"threads_started":1}
I20260926 20:24:27.855466 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:27.855013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.194:54233)
0926 20:24:27.855308 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:27.855372 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":81,"threads_started":1}
I20260926 20:24:28.044186 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:28.043731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.195:39197)
0926 20:24:28.044028 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:28.044091 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":82,"threads_started":1}
I20260926 20:24:28.706046 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:28.705549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.193:52961)
0926 20:24:28.705835 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:28.705944 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":74,"threads_started":1}
I20260926 20:24:28.856810 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:28.856305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.194:40557)
0926 20:24:28.856645 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:28.856709 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":120,"threads_started":1}
I20260926 20:24:29.045552 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:29.045068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.195:59327)
0926 20:24:29.045376 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:29.045448 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":81,"threads_started":1}
I20260926 20:24:29.707401 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:29.706878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.193:45669)
0926 20:24:29.707166 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:29.707228 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":85,"threads_started":1}
I20260926 20:24:29.858145 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:29.857679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.194:40231)
0926 20:24:29.857977 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:29.858045 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":80,"threads_started":1}
I20260926 20:24:30.046906 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:30.046475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.195:55209)
0926 20:24:30.046743 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:30.046802 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":96,"threads_started":1}
I20260926 20:24:30.150125 17449 raft_consensus.cc:482] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:24:30.150292 17449 raft_consensus.cc:504] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc16bee212ed4616bbf66ff7dc175799" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 38389 } } peers { permanent_uuid: "b90889308e1941e28b5eb97e45b1f913" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42889 } } peers { permanent_uuid: "e77b5a0d96c94986a49ba05cc9a068fe" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43847 } }
I20260926 20:24:30.150725 17449 leader_election.cc:288] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc16bee212ed4616bbf66ff7dc175799 (127.0.116.195:38389), b90889308e1941e28b5eb97e45b1f913 (127.0.116.194:42889)
W20260926 20:24:30.151619 17034 leader_election.cc:341] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc16bee212ed4616bbf66ff7dc175799 (127.0.116.195:38389): Not found: Tablet not found: 32362f05d9534bb2a797c837170d4f85
W20260926 20:24:30.151786 17034 leader_election.cc:341] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b90889308e1941e28b5eb97e45b1f913 (127.0.116.194:42889): Not found: Tablet not found: 32362f05d9534bb2a797c837170d4f85
I20260926 20:24:30.151849 17034 leader_election.cc:302] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e77b5a0d96c94986a49ba05cc9a068fe; no voters: b90889308e1941e28b5eb97e45b1f913, bc16bee212ed4616bbf66ff7dc175799
I20260926 20:24:30.152043 17449 raft_consensus.cc:2726] T 32362f05d9534bb2a797c837170d4f85 P e77b5a0d96c94986a49ba05cc9a068fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:24:30.708802 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:30.708330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.193:35049)
0926 20:24:30.708630 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:30.708690 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":107,"threads_started":1}
I20260926 20:24:30.859483 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:30.859032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.194:55859)
0926 20:24:30.859313 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:30.859386 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":84,"threads_started":1}
I20260926 20:24:31.048468 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:31.047864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.195:46481)
0926 20:24:31.048262 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:31.048342 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":105,"threads_started":1}
I20260926 20:24:31.710492 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:31.710024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.193:38969)
0926 20:24:31.710310 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:31.710393 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":76,"threads_started":1}
I20260926 20:24:31.860822 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:31.860382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.194:33485)
0926 20:24:31.860655 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:31.860718 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":81,"threads_started":1}
I20260926 20:24:32.049871 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:32.049240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.195:37205)
0926 20:24:32.049565 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:32.049630 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":103,"threads_started":1}
I20260926 20:24:32.711910 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:32.711345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.193:60653)
0926 20:24:32.711743 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:32.711812 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":95,"threads_started":1}
I20260926 20:24:32.862253 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:32.861775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.194:56717)
0926 20:24:32.862083 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:32.862154 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":86,"threads_started":1}
I20260926 20:24:33.051214 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:33.050753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.195:41289)
0926 20:24:33.051053 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:33.051116 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":85,"threads_started":1}
I20260926 20:24:33.713097 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:33.712666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.193:37701)
0926 20:24:33.712933 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:33.713000 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
I20260926 20:24:33.863430 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:33.862983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.194:47187)
0926 20:24:33.863253 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:33.863314 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
I20260926 20:24:34.052632 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:34.052140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.195:46619)
0926 20:24:34.052466 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:34.052531 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":91,"threads_started":1}
I20260926 20:24:34.714349 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:34.713888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.193:48855)
0926 20:24:34.714178 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:34.714244 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":83,"threads_started":1}
I20260926 20:24:34.864686 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:34.864241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.194:41847)
0926 20:24:34.864524 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:34.864587 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":75,"threads_started":1}
I20260926 20:24:35.054075 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:35.053551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.195:58957)
0926 20:24:35.053901 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:35.053972 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":92,"threads_started":1}
I20260926 20:24:35.715751 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:35.715227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.193:57483)
0926 20:24:35.715585 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:35.715653 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":95,"threads_started":1}
I20260926 20:24:35.866250 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:35.865630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.194:54907)
0926 20:24:35.866048 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:35.866110 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":115,"threads_started":1}
I20260926 20:24:36.055450 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:36.054951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.195:36233)
0926 20:24:36.055246 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:36.055311 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":94,"threads_started":1}
I20260926 20:24:36.717231 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:36.716820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.193:54849)
0926 20:24:36.717084 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:36.717136 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
I20260926 20:24:36.867786 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:36.867239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:45107 (local address 127.0.116.194:47863)
0926 20:24:36.867617 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:36.867681 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.254:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":105,"threads_started":1}
I20260926 20:24:37.037060 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 16978
I20260926 20:24:37.042784 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 17152
I20260926 20:24:37.048276 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 17286
2026-09-26T20:24:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59482 ms)
[----------] 30 tests from SecurityITest (243069 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:24:37 dist-test-slave-qsm9 krb5kdc[17472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:24:37 dist-test-slave-qsm9 krb5kdc[17472](info): set up 2 sockets
Sep 26 20:24:37 dist-test-slave-qsm9 krb5kdc[17472](info): commencing operation
krb5kdc: starting...
W20260926 20:24:38.115789 467 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:24:38 dist-test-slave-qsm9 krb5kdc[17472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:24:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:24:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:24:38.244000 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:46707
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:46707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:24:38.357447 17485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:38.357749 17485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:38.357791 17485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:38.361907 17485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:24:38.361972 17485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:38.361991 17485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:38.362007 17485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:24:38.362023 17485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:24:38.367319 17485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:46707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:46707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:38.368535 17485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:38.369947 17485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:38.379868 17491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:38.379953 17493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:38.379953 17490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:38.380089 17485 server_base.cc:1034] running on GCE node
I20260926 20:24:38.380793 17485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:38.382069 17485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:38.383322 17485 hybrid_clock.cc:648] HybridClock initialized: now 1790454278383281 us; error 60 us; skew 500 ppm
Sep 26 20:24:38 dist-test-slave-qsm9 krb5kdc[17472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:38.386452 17485 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:24:38.387595 17485 webserver.cc:460] Webserver started at http://127.0.116.254:37105/ using document root <none> and password file <none>
I20260926 20:24:38.388182 17485 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:38.388276 17485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:38.388526 17485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:38.390755 17485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "8620091be2b94e9880a7ad69172a68bf"
format_stamp: "Formatted at 2026-09-26 20:24:38 on dist-test-slave-qsm9"
server_key: "602f006c5a6aaf7b47883a291e9eded9"
server_key_iv: "228963ab4950a28fac572e73c2a186d2"
server_key_version: "encryptionkey@0"
I20260926 20:24:38.391337 17485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "8620091be2b94e9880a7ad69172a68bf"
format_stamp: "Formatted at 2026-09-26 20:24:38 on dist-test-slave-qsm9"
server_key: "602f006c5a6aaf7b47883a291e9eded9"
server_key_iv: "228963ab4950a28fac572e73c2a186d2"
server_key_version: "encryptionkey@0"
I20260926 20:24:38.395285 17485 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 20:24:38.397958 17500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:38.399180 17485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:24:38.399328 17485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "8620091be2b94e9880a7ad69172a68bf"
format_stamp: "Formatted at 2026-09-26 20:24:38 on dist-test-slave-qsm9"
server_key: "602f006c5a6aaf7b47883a291e9eded9"
server_key_iv: "228963ab4950a28fac572e73c2a186d2"
server_key_version: "encryptionkey@0"
I20260926 20:24:38.399474 17485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:38.426842 17485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:38.430727 17485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:38.430970 17485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:38.439088 17485 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:46707
I20260926 20:24:38.439117 17552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:46707 every 8 connection(s)
I20260926 20:24:38.440182 17485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:24:38.440913 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 17485
I20260926 20:24:38.441064 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:24:38.441371 467 external_mini_cluster.cc:1428] Setting key 4a052a46704085516da2100334b4f4f3
I20260926 20:24:38.443569 17553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:38.450280 17553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf: Bootstrap starting.
Sep 26 20:24:38 dist-test-slave-qsm9 krb5kdc[17472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:24:38.452749 17553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:38.453496 17553 log.cc:824] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:38.455430 17553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf: No bootstrap required, opened a new log
I20260926 20:24:38.456691 17556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:38.443011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50238 (local address 127.0.116.254:46707)
0926 20:24:38.443535 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:38.443556 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:38.443606 (+ 50us) server_negotiation.cc:407] Connection header received
0926 20:24:38.444310 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:38.444327 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:38.444657 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:38.444944 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:38.445948 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:38.447048 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:38.448303 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:38.448796 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:38.448932 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:38.451900 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:38.451906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:38.451920 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:38.451950 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:38.453804 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:38.454286 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:38.454291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:38.454296 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:38.454411 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:38.454701 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:38.454705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:38.454707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:38.455115 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:38.455321 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:38.455658 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 20:24:38.455980 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":139,"threads_started":1}
I20260926 20:24:38.459580 17553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8620091be2b94e9880a7ad69172a68bf" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46707 } }
I20260926 20:24:38.459838 17553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:38.459921 17553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8620091be2b94e9880a7ad69172a68bf, State: Initialized, Role: FOLLOWER
I20260926 20:24:38.460346 17553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8620091be2b94e9880a7ad69172a68bf" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46707 } }
I20260926 20:24:38.460500 17553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:24:38.460585 17553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:24:38.460703 17553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:38.461862 17553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8620091be2b94e9880a7ad69172a68bf" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46707 } }
I20260926 20:24:38.462273 17553 leader_election.cc:302] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8620091be2b94e9880a7ad69172a68bf; no voters:
I20260926 20:24:38.462662 17553 leader_election.cc:288] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:24:38.462808 17558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:24:38.463109 17558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf [term 1 LEADER]: Becoming Leader. State: Replica: 8620091be2b94e9880a7ad69172a68bf, State: Running, Role: LEADER
I20260926 20:24:38.463465 17558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8620091be2b94e9880a7ad69172a68bf" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46707 } }
I20260926 20:24:38.464042 17553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:24:38.465328 17559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8620091be2b94e9880a7ad69172a68bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8620091be2b94e9880a7ad69172a68bf" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46707 } } }
I20260926 20:24:38.465382 17560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8620091be2b94e9880a7ad69172a68bf. Latest consensus state: current_term: 1 leader_uuid: "8620091be2b94e9880a7ad69172a68bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8620091be2b94e9880a7ad69172a68bf" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46707 } } }
I20260926 20:24:38.465507 17560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:38.465454 17559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf [sys.catalog]: This master's current role is: LEADER
W20260926 20:24:38.469166 17573 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:24:38.469239 17573 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:24:38.469321 17568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:24:38.470130 17568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:24:38.475639 17568 catalog_manager.cc:1347] Generated new cluster ID: 848d4297e7da46dea45c9cfe392e92ce
I20260926 20:24:38.475732 17568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:24:38.488363 17568 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:24:38.489948 17568 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:24:38.495831 17568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf: Generated new TSK 0
I20260926 20:24:38.496697 17568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
I20260926 20:24:38.526810 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46707
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:24:38.641126 17579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:38.641477 17579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:38.641559 17579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:38.645826 17579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:38.645922 17579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:38.646018 17579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:24:38.651566 17579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:38.652746 17579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:38.653821 17579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:38.663872 17585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:38.663857 17584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:38.663894 17587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:38.663977 17579 server_base.cc:1034] running on GCE node
I20260926 20:24:38.664623 17579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:38.665400 17579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:38.666587 17579 hybrid_clock.cc:648] HybridClock initialized: now 1790454278666569 us; error 31 us; skew 500 ppm
Sep 26 20:24:38 dist-test-slave-qsm9 krb5kdc[17472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:38.669368 17579 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:24:38.670223 17579 webserver.cc:460] Webserver started at http://127.0.116.193:42277/ using document root <none> and password file <none>
I20260926 20:24:38.670789 17579 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:38.670877 17579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:38.671121 17579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:38.673261 17579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "1d6dfd4c7bc1466fa3a55e69da338271"
format_stamp: "Formatted at 2026-09-26 20:24:38 on dist-test-slave-qsm9"
server_key: "b887aad3dc6df6063ab2584ce954c101"
server_key_iv: "2d8ab08941ae974e67ce0b7100bd9fc2"
server_key_version: "encryptionkey@0"
I20260926 20:24:38.673856 17579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "1d6dfd4c7bc1466fa3a55e69da338271"
format_stamp: "Formatted at 2026-09-26 20:24:38 on dist-test-slave-qsm9"
server_key: "b887aad3dc6df6063ab2584ce954c101"
server_key_iv: "2d8ab08941ae974e67ce0b7100bd9fc2"
server_key_version: "encryptionkey@0"
I20260926 20:24:38.677541 17579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 20:24:38.679916 17594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:38.681181 17579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:24:38.681335 17579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "1d6dfd4c7bc1466fa3a55e69da338271"
format_stamp: "Formatted at 2026-09-26 20:24:38 on dist-test-slave-qsm9"
server_key: "b887aad3dc6df6063ab2584ce954c101"
server_key_iv: "2d8ab08941ae974e67ce0b7100bd9fc2"
server_key_version: "encryptionkey@0"
I20260926 20:24:38.681444 17579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:38.702620 17579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:38.705884 17579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:38.706094 17579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:38.706703 17579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:38.707608 17579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:24:38.707675 17579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:38.707737 17579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:24:38.707787 17579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:38.717512 17579 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:41595
I20260926 20:24:38.717535 17707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:41595 every 8 connection(s)
I20260926 20:24:38.718456 17579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:24:38.724598 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 17579
I20260926 20:24:38.724740 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:24:38.725050 467 external_mini_cluster.cc:1428] Setting key 92ad80f9f647dc2c10987266c37eeb2b
Sep 26 20:24:38 dist-test-slave-qsm9 krb5kdc[17472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:24:38.733208 17556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:38.720558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:52465 (local address 127.0.116.254:46707)
0926 20:24:38.720693 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:38.720697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:38.721619 (+ 922us) server_negotiation.cc:407] Connection header received
0926 20:24:38.722451 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:38.722454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:38.722509 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:38.722594 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:38.723854 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:38.724416 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:38.726051 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:38.726402 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:38.726547 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:38.729506 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:38.729509 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:38.729511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:38.729536 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:38.731088 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:38.731664 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:38.731668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:38.731670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:38.731756 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:38.732113 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:38.732117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:38.732119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:38.732301 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:38.732408 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:38.732910 (+ 502us) server_negotiation.cc:299] Negotiation successful
0926 20:24:38.733044 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:24:38.734038 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:38.720951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46707 (local address 127.0.116.193:52465)
0926 20:24:38.721420 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:38.721463 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:38.722232 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:38.722754 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:38.722760 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:38.723165 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:38.723649 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:38.723663 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:38.724562 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:38.724566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:38.725924 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:38.725933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:38.726590 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:38.726593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:38.726796 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:38.727536 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:24:38.727566 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:38.729305 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:38.731211 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:38.731218 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:38.731235 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:38.731552 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:38.731850 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:38.731854 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:38.731856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:38.732010 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:38.732407 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:38.732414 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:38.732669 (+ 255us) client_negotiation.cc:769] Sending connection context
0926 20:24:38.732857 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 20:24:38.733202 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":130,"threads_started":1}
WARNING: no policy specified for kudu/127.0.116.194@KRBTEST.COM; defaulting to no policy
I20260926 20:24:38.735342 17708 heartbeater.cc:347] Connected to a master server at 127.0.116.254:46707
I20260926 20:24:38.735646 17708 heartbeater.cc:464] Registering TS with master...
I20260926 20:24:38.736246 17708 heartbeater.cc:511] Master 127.0.116.254:46707 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.194@KRBTEST.COM" created.
I20260926 20:24:38.737850 17517 ts_manager.cc:194] Registered new tserver with Master: 1d6dfd4c7bc1466fa3a55e69da338271 (127.0.116.193:41595)
I20260926 20:24:38.739051 17517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:52465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
I20260926 20:24:38.754487 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46707
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:24:38.869411 17713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:38.869679 17713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:38.869724 17713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:38.873886 17713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:38.873945 17713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:38.874009 17713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:24:38.879166 17713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:38.880242 17713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:38.881286 17713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:38.891686 17721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:38.891795 17719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:38.891716 17718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:38.891891 17713 server_base.cc:1034] running on GCE node
I20260926 20:24:38.892374 17713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:38.893101 17713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:38.894327 17713 hybrid_clock.cc:648] HybridClock initialized: now 1790454278894276 us; error 63 us; skew 500 ppm
Sep 26 20:24:38 dist-test-slave-qsm9 krb5kdc[17472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:38.897378 17713 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:24:38.898313 17713 webserver.cc:460] Webserver started at http://127.0.116.194:36843/ using document root <none> and password file <none>
I20260926 20:24:38.898866 17713 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:38.898926 17713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:38.899092 17713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:38.901208 17713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "17076b6cdcbe45549a9fac48c8d9e85f"
format_stamp: "Formatted at 2026-09-26 20:24:38 on dist-test-slave-qsm9"
server_key: "bd82ea1b5d47475b47e41cd20ed955cf"
server_key_iv: "3a1492fb3cc4662887106c27c81862c5"
server_key_version: "encryptionkey@0"
I20260926 20:24:38.901769 17713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "17076b6cdcbe45549a9fac48c8d9e85f"
format_stamp: "Formatted at 2026-09-26 20:24:38 on dist-test-slave-qsm9"
server_key: "bd82ea1b5d47475b47e41cd20ed955cf"
server_key_iv: "3a1492fb3cc4662887106c27c81862c5"
server_key_version: "encryptionkey@0"
I20260926 20:24:38.905586 17713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:24:38.908140 17728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:38.909435 17713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:24:38.909555 17713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "17076b6cdcbe45549a9fac48c8d9e85f"
format_stamp: "Formatted at 2026-09-26 20:24:38 on dist-test-slave-qsm9"
server_key: "bd82ea1b5d47475b47e41cd20ed955cf"
server_key_iv: "3a1492fb3cc4662887106c27c81862c5"
server_key_version: "encryptionkey@0"
I20260926 20:24:38.909654 17713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:38.928469 17713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:38.931792 17713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:38.931974 17713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:38.932556 17713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:38.933393 17713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:24:38.933449 17713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:38.933494 17713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:24:38.933519 17713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:38.943562 17841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:45793 every 8 connection(s)
I20260926 20:24:38.943567 17713 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:45793
I20260926 20:24:38.944591 17713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:24:38.951848 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 17713
I20260926 20:24:38.951982 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:24:38.952280 467 external_mini_cluster.cc:1428] Setting key 97a8c031776d6d716dce36f824f37fe5
Sep 26 20:24:38 dist-test-slave-qsm9 krb5kdc[17472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:24:38.959192 17556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:38.946502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:40123 (local address 127.0.116.254:46707)
0926 20:24:38.946683 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:38.946688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:38.947913 (+ 1225us) server_negotiation.cc:407] Connection header received
0926 20:24:38.948818 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:38.948820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:38.948870 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:38.949003 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:38.950220 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:38.950740 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:38.952123 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:38.952339 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:38.952466 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:38.955400 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:38.955404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:38.955406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:38.955430 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:38.957128 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:38.957650 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:38.957654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:38.957656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:38.957737 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:38.958077 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:38.958082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:38.958084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:38.958266 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:38.958375 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:38.958887 (+ 512us) server_negotiation.cc:299] Negotiation successful
0926 20:24:38.959028 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 20:24:38.960134 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:38.946945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46707 (local address 127.0.116.194:40123)
0926 20:24:38.947742 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:38.947786 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:38.948582 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:38.949175 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:38.949182 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:38.949577 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:38.950080 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:38.950095 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:38.950856 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:38.950860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:38.951991 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:38.952004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:38.952558 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:38.952564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:38.952765 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:38.953462 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:24:38.953522 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:38.955228 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:38.957248 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:38.957254 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:38.957271 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:38.957544 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:38.957831 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:38.957835 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:38.957837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:38.957980 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:38.958373 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:38.958380 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:38.958652 (+ 272us) client_negotiation.cc:769] Sending connection context
0926 20:24:38.958847 (+ 195us) client_negotiation.cc:240] Negotiation successful
0926 20:24:38.959191 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":127,"threads_started":1}
I20260926 20:24:38.961400 17842 heartbeater.cc:347] Connected to a master server at 127.0.116.254:46707
I20260926 20:24:38.961680 17842 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.116.195@KRBTEST.COM; defaulting to no policy
I20260926 20:24:38.962190 17842 heartbeater.cc:511] Master 127.0.116.254:46707 requested a full tablet report, sending...
I20260926 20:24:38.963362 17517 ts_manager.cc:194] Registered new tserver with Master: 17076b6cdcbe45549a9fac48c8d9e85f (127.0.116.194:45793)
I20260926 20:24:38.964247 17517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:40123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
I20260926 20:24:38.983072 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46707
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:24:39.095767 17847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:39.096055 17847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:39.096132 17847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:39.100924 17847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:39.100986 17847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:39.101078 17847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:24:39.106817 17847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:39.108117 17847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:39.109189 17847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:39.119230 17855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:39.119288 17853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:39.119233 17852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:39.119640 17847 server_base.cc:1034] running on GCE node
I20260926 20:24:39.120043 17847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:39.120745 17847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:39.121945 17847 hybrid_clock.cc:648] HybridClock initialized: now 1790454279121904 us; error 51 us; skew 500 ppm
Sep 26 20:24:39 dist-test-slave-qsm9 krb5kdc[17472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:39.124938 17847 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:24:39.125893 17847 webserver.cc:460] Webserver started at http://127.0.116.195:42583/ using document root <none> and password file <none>
I20260926 20:24:39.126507 17847 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:39.126606 17847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:39.126844 17847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:39.129038 17847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "1eca39461fe24b40a759dca67999e62d"
format_stamp: "Formatted at 2026-09-26 20:24:39 on dist-test-slave-qsm9"
server_key: "4451c298fdc523588c92253a2aa371c8"
server_key_iv: "01f7b9d61128b43a5f2d197198521be3"
server_key_version: "encryptionkey@0"
I20260926 20:24:39.129613 17847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "1eca39461fe24b40a759dca67999e62d"
format_stamp: "Formatted at 2026-09-26 20:24:39 on dist-test-slave-qsm9"
server_key: "4451c298fdc523588c92253a2aa371c8"
server_key_iv: "01f7b9d61128b43a5f2d197198521be3"
server_key_version: "encryptionkey@0"
I20260926 20:24:39.133514 17847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:24:39.136051 17862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:39.137298 17847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260926 20:24:39.137454 17847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "1eca39461fe24b40a759dca67999e62d"
format_stamp: "Formatted at 2026-09-26 20:24:39 on dist-test-slave-qsm9"
server_key: "4451c298fdc523588c92253a2aa371c8"
server_key_iv: "01f7b9d61128b43a5f2d197198521be3"
server_key_version: "encryptionkey@0"
I20260926 20:24:39.137575 17847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:39.146610 17847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:39.149765 17847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:39.149971 17847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:39.150563 17847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:39.151384 17847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:24:39.151463 17847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:39.151554 17847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:24:39.151611 17847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:39.161157 17847 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:35761
I20260926 20:24:39.161177 17975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:35761 every 8 connection(s)
I20260926 20:24:39.162441 17847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:24:39.170538 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 17847
I20260926 20:24:39.170697 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:24:39.171096 467 external_mini_cluster.cc:1428] Setting key 6e7be8b2d7ef0972a6b80f1000895be2
Sep 26 20:24:39 dist-test-slave-qsm9 krb5kdc[17472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:24:39.177497 17556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:39.164349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:35163 (local address 127.0.116.254:46707)
0926 20:24:39.164539 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:39.164543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:39.165875 (+ 1332us) server_negotiation.cc:407] Connection header received
0926 20:24:39.166821 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:39.166823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:39.166873 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:39.166993 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:39.168424 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.168956 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.170232 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.170443 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.170523 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:39.173730 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:39.173733 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:39.173736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:39.173765 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:39.175300 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:39.175885 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:39.175894 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:39.175898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:39.175976 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:39.176342 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:39.176345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:39.176347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:39.176544 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:39.176647 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:39.177200 (+ 553us) server_negotiation.cc:299] Negotiation successful
0926 20:24:39.177323 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 20:24:39.178349 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:39.164836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46707 (local address 127.0.116.195:35163)
0926 20:24:39.165709 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:39.165758 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:39.166614 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:39.167170 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:39.167177 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:39.167698 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:39.168258 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:39.168272 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.169077 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.169080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:39.170113 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:39.170124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.170641 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.170647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:39.170898 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:39.171701 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:24:39.171718 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:39.173558 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:39.175425 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:39.175433 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:39.175450 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:39.175771 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:39.176075 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:39.176078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:39.176080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:39.176234 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:39.176660 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:39.176668 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:39.176957 (+ 289us) client_negotiation.cc:769] Sending connection context
0926 20:24:39.177148 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 20:24:39.177488 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":113,"threads_started":1}
I20260926 20:24:39.179663 17976 heartbeater.cc:347] Connected to a master server at 127.0.116.254:46707
I20260926 20:24:39.179991 17976 heartbeater.cc:464] Registering TS with master...
I20260926 20:24:39.180562 17976 heartbeater.cc:511] Master 127.0.116.254:46707 requested a full tablet report, sending...
I20260926 20:24:39.181725 17517 ts_manager.cc:194] Registered new tserver with Master: 1eca39461fe24b40a759dca67999e62d (127.0.116.195:35761)
I20260926 20:24:39.182317 17517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:35163
I20260926 20:24:39.185941 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:24:39.196748 17556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:39.188111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50240 (local address 127.0.116.254:46707)
0926 20:24:39.188264 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:39.188270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:39.188488 (+ 218us) server_negotiation.cc:407] Connection header received
0926 20:24:39.188653 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:39.188656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:39.188704 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:39.188805 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:39.189589 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.190128 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.191028 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.191276 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.191400 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:39.192947 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:39.192951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:39.192954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:39.192992 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:39.194613 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:39.195107 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:39.195111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:39.195113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:39.195206 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:39.195624 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:39.195627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:39.195628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:39.195932 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:39.196063 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:39.196365 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 20:24:39.196549 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 20:24:39.200424 17517 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:24:39.203702 17517 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:24:39.219588 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:39.213354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:35761 (local address 127.0.0.1:54468)
0926 20:24:39.214290 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:39.214371 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:39.214534 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:39.215200 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:39.215208 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:39.215232 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:39.215376 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:39.215383 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.216697 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.216703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:39.217953 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:39.217963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.218958 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.218962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:39.219071 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:39.219115 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 20:24:39.219240 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 20:24:39.219380 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260926 20:24:39.219694 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:39.213702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54468 (local address 127.0.116.195:35761)
0926 20:24:39.214224 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:39.214236 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:39.214491 (+ 255us) server_negotiation.cc:407] Connection header received
0926 20:24:39.214774 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:39.214783 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:39.215042 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:39.215195 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:39.215536 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.216559 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.218080 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.218822 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.218930 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:39.219084 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:39.219291 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 20:24:39.219400 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":156,"threads_started":1}
I20260926 20:24:39.220155 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:39.213987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:45793 (local address 127.0.0.1:40088)
0926 20:24:39.214991 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:39.215002 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:39.215139 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:39.215867 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:39.215870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:39.215893 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:39.216005 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:39.216011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.217323 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.217340 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:39.218460 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:39.218470 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.219671 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.219675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:39.219832 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:39.219862 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 20:24:39.219937 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:24:39.220014 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260926 20:24:39.220505 17992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:39.214372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.0.116.194:45793)
0926 20:24:39.214749 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:39.214760 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:39.215073 (+ 313us) server_negotiation.cc:407] Connection header received
0926 20:24:39.215278 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:39.215284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:39.215518 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:39.215642 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:39.216134 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.217192 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.218590 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.219343 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.219434 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:39.219605 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:39.219990 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 20:24:39.220092 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":166,"threads_started":1}
I20260926 20:24:39.220872 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:39.213517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.0.116.193:41595)
0926 20:24:39.214414 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:39.214439 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:39.215020 (+ 581us) server_negotiation.cc:407] Connection header received
0926 20:24:39.215138 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:39.215145 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:39.215338 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:39.215534 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:39.215809 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.216993 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.218553 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.219363 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.219466 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:39.219626 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:39.220629 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0926 20:24:39.220729 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":69,"threads_started":1}
I20260926 20:24:39.221251 17991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:39.213354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:41595 (local address 127.0.0.1:55262)
0926 20:24:39.214900 (+ 1546us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:39.214917 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:39.215022 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:39.215484 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:39.215487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:39.215545 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:39.215672 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:39.215679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.217143 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.217147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:39.218404 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:39.218415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.219693 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.219697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:39.220433 (+ 736us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:39.220470 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 20:24:39.221079 (+ 609us) client_negotiation.cc:240] Negotiation successful
0926 20:24:39.221157 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1472,"mutex_wait_us":632,"thread_start_us":114,"threads_started":1}
I20260926 20:24:39.223304 17776 tablet_service.cc:1467] Processing CreateTablet for tablet 975dae2bb06e4ac2b82f07a3d660f3cc (DEFAULT_TABLE table=test-table [id=6f379170b50f42f6b7a93d521e8c7f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:24:39.223197 17910 tablet_service.cc:1467] Processing CreateTablet for tablet 975dae2bb06e4ac2b82f07a3d660f3cc (DEFAULT_TABLE table=test-table [id=6f379170b50f42f6b7a93d521e8c7f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:24:39.224296 17776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975dae2bb06e4ac2b82f07a3d660f3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:39.223935 17642 tablet_service.cc:1467] Processing CreateTablet for tablet 975dae2bb06e4ac2b82f07a3d660f3cc (DEFAULT_TABLE table=test-table [id=6f379170b50f42f6b7a93d521e8c7f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:24:39.224344 17910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975dae2bb06e4ac2b82f07a3d660f3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:39.224869 17642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975dae2bb06e4ac2b82f07a3d660f3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:39.230387 17993 tablet_bootstrap.cc:492] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f: Bootstrap starting.
I20260926 20:24:39.232630 17994 tablet_bootstrap.cc:492] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1eca39461fe24b40a759dca67999e62d: Bootstrap starting.
I20260926 20:24:39.232785 17995 tablet_bootstrap.cc:492] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1d6dfd4c7bc1466fa3a55e69da338271: Bootstrap starting.
I20260926 20:24:39.233152 17993 tablet_bootstrap.cc:654] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:39.234584 17993 log.cc:824] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:39.234606 17994 tablet_bootstrap.cc:654] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1eca39461fe24b40a759dca67999e62d: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:39.235086 17995 tablet_bootstrap.cc:654] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1d6dfd4c7bc1466fa3a55e69da338271: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:39.235865 17994 log.cc:824] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1eca39461fe24b40a759dca67999e62d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:39.235937 17995 log.cc:824] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1d6dfd4c7bc1466fa3a55e69da338271: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:39.237015 17993 tablet_bootstrap.cc:492] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f: No bootstrap required, opened a new log
I20260926 20:24:39.237269 17993 ts_tablet_manager.cc:1397] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 20:24:39.237792 17995 tablet_bootstrap.cc:492] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1d6dfd4c7bc1466fa3a55e69da338271: No bootstrap required, opened a new log
I20260926 20:24:39.237790 17994 tablet_bootstrap.cc:492] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1eca39461fe24b40a759dca67999e62d: No bootstrap required, opened a new log
I20260926 20:24:39.237979 17995 ts_tablet_manager.cc:1397] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1d6dfd4c7bc1466fa3a55e69da338271: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:24:39.237979 17994 ts_tablet_manager.cc:1397] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1eca39461fe24b40a759dca67999e62d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:24:39.240424 17993 raft_consensus.cc:348] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dfd4c7bc1466fa3a55e69da338271" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 41595 } } peers { permanent_uuid: "1eca39461fe24b40a759dca67999e62d" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35761 } } peers { permanent_uuid: "17076b6cdcbe45549a9fac48c8d9e85f" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45793 } }
I20260926 20:24:39.240703 17993 raft_consensus.cc:374] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:39.240782 17993 raft_consensus.cc:729] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17076b6cdcbe45549a9fac48c8d9e85f, State: Initialized, Role: FOLLOWER
I20260926 20:24:39.241358 17993 consensus_queue.cc:260] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dfd4c7bc1466fa3a55e69da338271" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 41595 } } peers { permanent_uuid: "1eca39461fe24b40a759dca67999e62d" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35761 } } peers { permanent_uuid: "17076b6cdcbe45549a9fac48c8d9e85f" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45793 } }
I20260926 20:24:39.242132 17842 heartbeater.cc:503] Master 127.0.116.254:46707 was elected leader, sending a full tablet report...
I20260926 20:24:39.242061 17994 raft_consensus.cc:348] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1eca39461fe24b40a759dca67999e62d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dfd4c7bc1466fa3a55e69da338271" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 41595 } } peers { permanent_uuid: "1eca39461fe24b40a759dca67999e62d" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35761 } } peers { permanent_uuid: "17076b6cdcbe45549a9fac48c8d9e85f" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45793 } }
I20260926 20:24:39.242318 17994 raft_consensus.cc:374] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1eca39461fe24b40a759dca67999e62d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:39.242364 17994 raft_consensus.cc:729] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1eca39461fe24b40a759dca67999e62d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eca39461fe24b40a759dca67999e62d, State: Initialized, Role: FOLLOWER
I20260926 20:24:39.242471 17993 ts_tablet_manager.cc:1428] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:24:39.242792 17994 consensus_queue.cc:260] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1eca39461fe24b40a759dca67999e62d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dfd4c7bc1466fa3a55e69da338271" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 41595 } } peers { permanent_uuid: "1eca39461fe24b40a759dca67999e62d" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35761 } } peers { permanent_uuid: "17076b6cdcbe45549a9fac48c8d9e85f" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45793 } }
I20260926 20:24:39.242893 17995 raft_consensus.cc:348] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1d6dfd4c7bc1466fa3a55e69da338271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dfd4c7bc1466fa3a55e69da338271" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 41595 } } peers { permanent_uuid: "1eca39461fe24b40a759dca67999e62d" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35761 } } peers { permanent_uuid: "17076b6cdcbe45549a9fac48c8d9e85f" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45793 } }
I20260926 20:24:39.243191 17995 raft_consensus.cc:374] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1d6dfd4c7bc1466fa3a55e69da338271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:39.243307 17995 raft_consensus.cc:729] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1d6dfd4c7bc1466fa3a55e69da338271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d6dfd4c7bc1466fa3a55e69da338271, State: Initialized, Role: FOLLOWER
I20260926 20:24:39.243848 17995 consensus_queue.cc:260] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1d6dfd4c7bc1466fa3a55e69da338271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dfd4c7bc1466fa3a55e69da338271" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 41595 } } peers { permanent_uuid: "1eca39461fe24b40a759dca67999e62d" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35761 } } peers { permanent_uuid: "17076b6cdcbe45549a9fac48c8d9e85f" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45793 } }
I20260926 20:24:39.243911 17994 ts_tablet_manager.cc:1428] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1eca39461fe24b40a759dca67999e62d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 20:24:39.244318 17976 heartbeater.cc:503] Master 127.0.116.254:46707 was elected leader, sending a full tablet report...
I20260926 20:24:39.244884 17995 ts_tablet_manager.cc:1428] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1d6dfd4c7bc1466fa3a55e69da338271: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 20:24:39.245239 17708 heartbeater.cc:503] Master 127.0.116.254:46707 was elected leader, sending a full tablet report...
W20260926 20:24:39.413450 17977 tablet.cc:2369] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1eca39461fe24b40a759dca67999e62d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:24:39.445902 17843 tablet.cc:2369] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:24:39.453037 17999 raft_consensus.cc:482] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:24:39.453267 17999 raft_consensus.cc:504] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dfd4c7bc1466fa3a55e69da338271" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 41595 } } peers { permanent_uuid: "1eca39461fe24b40a759dca67999e62d" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35761 } } peers { permanent_uuid: "17076b6cdcbe45549a9fac48c8d9e85f" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45793 } }
I20260926 20:24:39.454684 17999 leader_election.cc:288] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d6dfd4c7bc1466fa3a55e69da338271 (127.0.116.193:41595), 1eca39461fe24b40a759dca67999e62d (127.0.116.195:35761)
I20260926 20:24:39.458730 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:39.454939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:41595 (local address 127.0.116.194:48593)
0926 20:24:39.455207 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:39.455227 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:39.455344 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:39.455667 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:39.455670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:39.455691 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:39.455831 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:39.455838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.456728 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.456731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:39.457597 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:39.457612 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.458292 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.458296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:39.458434 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:39.458459 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:24:39.458538 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 20:24:39.458618 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182}
I20260926 20:24:39.458853 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:39.455095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:48593 (local address 127.0.116.193:41595)
0926 20:24:39.455228 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:39.455233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:39.455313 (+ 80us) server_negotiation.cc:407] Connection header received
0926 20:24:39.455481 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:39.455484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:39.455558 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:39.455639 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:39.455968 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.456612 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.457730 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.458163 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.458256 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:39.458295 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:39.458599 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 20:24:39.458686 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:24:39.459100 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:39.454982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:35761 (local address 127.0.116.194:37031)
0926 20:24:39.455385 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:39.455423 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:39.455547 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:39.455878 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:39.455922 (+ 44us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:39.455948 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:39.456067 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:39.456073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.456841 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.456889 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:39.457710 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:39.457722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.458582 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.458586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:39.458706 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:39.458729 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:24:39.458885 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 20:24:39.458963 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":93,"threads_started":1}
I20260926 20:24:39.459197 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:39.455467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:37031 (local address 127.0.116.195:35761)
0926 20:24:39.455633 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:39.455638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:39.455652 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:24:39.455700 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:39.455702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:39.455744 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:39.455828 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:39.456191 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.456731 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.457857 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.458348 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.458470 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:39.458535 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:39.459003 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 20:24:39.459088 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 20:24:39.459627 17662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975dae2bb06e4ac2b82f07a3d660f3cc" candidate_uuid: "17076b6cdcbe45549a9fac48c8d9e85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d6dfd4c7bc1466fa3a55e69da338271" is_pre_election: true
I20260926 20:24:39.459915 17930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975dae2bb06e4ac2b82f07a3d660f3cc" candidate_uuid: "17076b6cdcbe45549a9fac48c8d9e85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca39461fe24b40a759dca67999e62d" is_pre_election: true
I20260926 20:24:39.460050 17662 raft_consensus.cc:2454] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1d6dfd4c7bc1466fa3a55e69da338271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17076b6cdcbe45549a9fac48c8d9e85f in term 0.
I20260926 20:24:39.460193 17930 raft_consensus.cc:2454] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1eca39461fe24b40a759dca67999e62d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17076b6cdcbe45549a9fac48c8d9e85f in term 0.
I20260926 20:24:39.460685 17729 leader_election.cc:302] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17076b6cdcbe45549a9fac48c8d9e85f, 1d6dfd4c7bc1466fa3a55e69da338271, 1eca39461fe24b40a759dca67999e62d; no voters:
I20260926 20:24:39.461016 17999 raft_consensus.cc:2781] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:24:39.461120 17999 raft_consensus.cc:482] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:24:39.461202 17999 raft_consensus.cc:3037] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:39.462260 17999 raft_consensus.cc:504] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dfd4c7bc1466fa3a55e69da338271" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 41595 } } peers { permanent_uuid: "1eca39461fe24b40a759dca67999e62d" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35761 } } peers { permanent_uuid: "17076b6cdcbe45549a9fac48c8d9e85f" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45793 } }
I20260926 20:24:39.462620 17999 leader_election.cc:288] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f [CANDIDATE]: Term 1 election: Requested vote from peers 1d6dfd4c7bc1466fa3a55e69da338271 (127.0.116.193:41595), 1eca39461fe24b40a759dca67999e62d (127.0.116.195:35761)
I20260926 20:24:39.463027 17662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975dae2bb06e4ac2b82f07a3d660f3cc" candidate_uuid: "17076b6cdcbe45549a9fac48c8d9e85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d6dfd4c7bc1466fa3a55e69da338271"
I20260926 20:24:39.463176 17930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975dae2bb06e4ac2b82f07a3d660f3cc" candidate_uuid: "17076b6cdcbe45549a9fac48c8d9e85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eca39461fe24b40a759dca67999e62d"
I20260926 20:24:39.463182 17662 raft_consensus.cc:3037] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1d6dfd4c7bc1466fa3a55e69da338271 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:39.463315 17930 raft_consensus.cc:3037] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1eca39461fe24b40a759dca67999e62d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:39.464675 17662 raft_consensus.cc:2454] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1d6dfd4c7bc1466fa3a55e69da338271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17076b6cdcbe45549a9fac48c8d9e85f in term 1.
I20260926 20:24:39.464675 17930 raft_consensus.cc:2454] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1eca39461fe24b40a759dca67999e62d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17076b6cdcbe45549a9fac48c8d9e85f in term 1.
I20260926 20:24:39.464983 17729 leader_election.cc:302] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17076b6cdcbe45549a9fac48c8d9e85f, 1d6dfd4c7bc1466fa3a55e69da338271; no voters:
I20260926 20:24:39.465173 17999 raft_consensus.cc:2781] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:24:39.465505 17999 raft_consensus.cc:686] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f [term 1 LEADER]: Becoming Leader. State: Replica: 17076b6cdcbe45549a9fac48c8d9e85f, State: Running, Role: LEADER
I20260926 20:24:39.465862 17999 consensus_queue.cc:237] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dfd4c7bc1466fa3a55e69da338271" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 41595 } } peers { permanent_uuid: "1eca39461fe24b40a759dca67999e62d" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35761 } } peers { permanent_uuid: "17076b6cdcbe45549a9fac48c8d9e85f" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45793 } }
I20260926 20:24:39.469209 17516 catalog_manager.cc:5495] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f reported cstate change: term changed from 0 to 1, leader changed from <none> to 17076b6cdcbe45549a9fac48c8d9e85f (127.0.116.194). New cstate: current_term: 1 leader_uuid: "17076b6cdcbe45549a9fac48c8d9e85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6dfd4c7bc1466fa3a55e69da338271" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 41595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eca39461fe24b40a759dca67999e62d" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17076b6cdcbe45549a9fac48c8d9e85f" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 45793 } health_report { overall_health: HEALTHY } } }
W20260926 20:24:39.469799 17709 tablet.cc:2369] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1d6dfd4c7bc1466fa3a55e69da338271: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:24:39.520996 17992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:39.517295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40090 (local address 127.0.116.194:45793)
0926 20:24:39.517421 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:39.517425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:39.517439 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:24:39.517512 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:39.517514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:39.517568 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:39.517663 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:24:39.517948 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.518476 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.519431 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.519721 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.519839 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:39.520175 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:24:39.520644 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:24:39.520747 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:39.520805 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 20:24:39.520865 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:24:39.527135 17930 raft_consensus.cc:1261] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1eca39461fe24b40a759dca67999e62d [term 1 FOLLOWER]: Refusing update from remote peer 17076b6cdcbe45549a9fac48c8d9e85f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:24:39.527136 17662 raft_consensus.cc:1261] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1d6dfd4c7bc1466fa3a55e69da338271 [term 1 FOLLOWER]: Refusing update from remote peer 17076b6cdcbe45549a9fac48c8d9e85f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:24:39.527956 18003 consensus_queue.cc:1035] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eca39461fe24b40a759dca67999e62d" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:24:39.528147 17999 consensus_queue.cc:1035] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d6dfd4c7bc1466fa3a55e69da338271" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 41595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:24:39.535478 18007 mvcc.cc:205] Tried to move back new op lower bound from 7333700728935202816 to 7333700728695033856. Current Snapshot: MvccSnapshot[applied={T|T < 7333700728935202816}]
I20260926 20:24:39.544672 17517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:24:39.544834 17517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:24:39.547263 17517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8620091be2b94e9880a7ad69172a68bf: Sending DeleteTablet for 3 replicas of tablet 975dae2bb06e4ac2b82f07a3d660f3cc
I20260926 20:24:39.548143 17910 tablet_service.cc:1514] Processing DeleteTablet for tablet 975dae2bb06e4ac2b82f07a3d660f3cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:24:39 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:54468
I20260926 20:24:39.548144 17642 tablet_service.cc:1514] Processing DeleteTablet for tablet 975dae2bb06e4ac2b82f07a3d660f3cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:24:39 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:55262
I20260926 20:24:39.548696 18016 tablet_replica.cc:330] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1eca39461fe24b40a759dca67999e62d: stopping tablet replica
I20260926 20:24:39.548755 17776 tablet_service.cc:1514] Processing DeleteTablet for tablet 975dae2bb06e4ac2b82f07a3d660f3cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:24:39 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:40088
I20260926 20:24:39.549068 18016 raft_consensus.cc:2229] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1eca39461fe24b40a759dca67999e62d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:24:39.549320 18018 tablet_replica.cc:330] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f: stopping tablet replica
I20260926 20:24:39.549431 18016 raft_consensus.cc:2258] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1eca39461fe24b40a759dca67999e62d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:24:39.549638 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 17579
I20260926 20:24:39.549692 18018 raft_consensus.cc:2229] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:24:39.549824 18017 tablet_replica.cc:330] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1d6dfd4c7bc1466fa3a55e69da338271: stopping tablet replica
I20260926 20:24:39.550591 18018 raft_consensus.cc:2258] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:24:39.551905 18018 ts_tablet_manager.cc:1905] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:24:39.551990 18016 ts_tablet_manager.cc:1905] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1eca39461fe24b40a759dca67999e62d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:24:39.554280 18018 ts_tablet_manager.cc:1918] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:24:39.554354 18018 log.cc:1196] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/975dae2bb06e4ac2b82f07a3d660f3cc
I20260926 20:24:39.554584 18018 ts_tablet_manager.cc:1939] T 975dae2bb06e4ac2b82f07a3d660f3cc P 17076b6cdcbe45549a9fac48c8d9e85f: Deleting consensus metadata
I20260926 20:24:39.555393 18016 ts_tablet_manager.cc:1918] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1eca39461fe24b40a759dca67999e62d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:24:39.555486 18016 log.cc:1196] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1eca39461fe24b40a759dca67999e62d: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/975dae2bb06e4ac2b82f07a3d660f3cc
I20260926 20:24:39.555666 17502 catalog_manager.cc:4841] TS 17076b6cdcbe45549a9fac48c8d9e85f (127.0.116.194:45793): tablet 975dae2bb06e4ac2b82f07a3d660f3cc (table test-table [id=6f379170b50f42f6b7a93d521e8c7f9b]) successfully deleted
I20260926 20:24:39.555856 18016 ts_tablet_manager.cc:1939] T 975dae2bb06e4ac2b82f07a3d660f3cc P 1eca39461fe24b40a759dca67999e62d: Deleting consensus metadata
I20260926 20:24:39.557094 17503 catalog_manager.cc:4841] TS 1eca39461fe24b40a759dca67999e62d (127.0.116.195:35761): tablet 975dae2bb06e4ac2b82f07a3d660f3cc (table test-table [id=6f379170b50f42f6b7a93d521e8c7f9b]) successfully deleted
W20260926 20:24:39.558980 17501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.116.193:41595 (error 108)
I20260926 20:24:39.559228 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 17713
I20260926 20:24:39.564078 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 17847
I20260926 20:24:39.564455 17991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:39.564092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:41595 (local address 127.0.0.1:55272)
0926 20:24:39.564250 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:39.564333 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.193:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 20:24:39.565274 17501 catalog_manager.cc:4584] TS 1d6dfd4c7bc1466fa3a55e69da338271 (127.0.116.193:41595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 975dae2bb06e4ac2b82f07a3d660f3cc: Network error: Client connection negotiation failed: client connection to 127.0.116.193:41595: connect: Connection refused (error 111)
I20260926 20:24:39.569823 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 17485
2026-09-26T20:24:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2520 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T20:24:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:24:39Z Disabled control of system clock
I20260926 20:24:39.601202 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:44825
--webserver_interface=127.0.116.254
--webserver_port=0
--builtin_ntp_servers=127.0.116.212:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:44825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:24:39.717514 18023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:39.717803 18023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:39.717841 18023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:39.721983 18023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:24:39.722067 18023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:39.722083 18023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:39.722100 18023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:24:39.722116 18023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:24:39.727234 18023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:44825
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:44825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:39.728457 18023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:39.729919 18023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:39.738940 18028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:39.738940 18029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:39.739110 18023 server_base.cc:1034] running on GCE node
W20260926 20:24:39.739360 18031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:39.740010 18023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:39.741194 18023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:39.742399 18023 hybrid_clock.cc:648] HybridClock initialized: now 1790454279742380 us; error 33 us; skew 500 ppm
I20260926 20:24:39.744475 18023 webserver.cc:460] Webserver started at http://127.0.116.254:40845/ using document root <none> and password file <none>
I20260926 20:24:39.745091 18023 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:39.745158 18023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:39.745409 18023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:39.747686 18023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "14e6dedc94be4638bd2a0019c5f94c09"
format_stamp: "Formatted at 2026-09-26 20:24:39 on dist-test-slave-qsm9"
server_key: "674ef49373490e4b735b355d07db05f7"
server_key_iv: "a6625f37bca8383b07b523a78a026f8b"
server_key_version: "encryptionkey@0"
I20260926 20:24:39.748301 18023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "14e6dedc94be4638bd2a0019c5f94c09"
format_stamp: "Formatted at 2026-09-26 20:24:39 on dist-test-slave-qsm9"
server_key: "674ef49373490e4b735b355d07db05f7"
server_key_iv: "a6625f37bca8383b07b523a78a026f8b"
server_key_version: "encryptionkey@0"
I20260926 20:24:39.752341 18023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:24:39.755105 18037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:39.756670 18023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:24:39.756826 18023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "14e6dedc94be4638bd2a0019c5f94c09"
format_stamp: "Formatted at 2026-09-26 20:24:39 on dist-test-slave-qsm9"
server_key: "674ef49373490e4b735b355d07db05f7"
server_key_iv: "a6625f37bca8383b07b523a78a026f8b"
server_key_version: "encryptionkey@0"
I20260926 20:24:39.756963 18023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:39.775449 18023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:39.776261 18023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:39.776481 18023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:39.783396 18023 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:44825
I20260926 20:24:39.783454 18089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:44825 every 8 connection(s)
I20260926 20:24:39.784473 18023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:24:39.787717 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 18023
I20260926 20:24:39.787842 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:24:39.788069 18090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:39.788174 467 external_mini_cluster.cc:1428] Setting key 4d64deb95963246159711f772df12fdd
I20260926 20:24:39.795889 18090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09: Bootstrap starting.
I20260926 20:24:39.798597 18090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:39.799474 18090 log.cc:824] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:39.799667 18093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:39.789701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59816 (local address 127.0.116.254:44825)
0926 20:24:39.790420 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:39.790452 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:39.790496 (+ 44us) server_negotiation.cc:407] Connection header received
0926 20:24:39.791768 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:39.791795 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:39.792334 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:39.792732 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:39.793180 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.794765 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.795766 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:39.796465 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:39.796621 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:39.797535 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:39.797542 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:39.797558 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:39.798052 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:39.798129 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:39.798155 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:39.798252 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:39.798385 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 20:24:39.798674 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":299,"thread_start_us":127,"threads_started":1}
I20260926 20:24:39.801735 18090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09: No bootstrap required, opened a new log
I20260926 20:24:39.805523 18090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14e6dedc94be4638bd2a0019c5f94c09" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44825 } }
I20260926 20:24:39.805824 18090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:39.805886 18090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e6dedc94be4638bd2a0019c5f94c09, State: Initialized, Role: FOLLOWER
I20260926 20:24:39.806357 18090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14e6dedc94be4638bd2a0019c5f94c09" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44825 } }
I20260926 20:24:39.806499 18090 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:24:39.806584 18090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:24:39.806681 18090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:39.807717 18090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14e6dedc94be4638bd2a0019c5f94c09" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44825 } }
I20260926 20:24:39.808056 18090 leader_election.cc:302] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14e6dedc94be4638bd2a0019c5f94c09; no voters:
I20260926 20:24:39.808358 18090 leader_election.cc:288] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:24:39.808561 18095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:24:39.808856 18095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09 [term 1 LEADER]: Becoming Leader. State: Replica: 14e6dedc94be4638bd2a0019c5f94c09, State: Running, Role: LEADER
I20260926 20:24:39.809156 18095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14e6dedc94be4638bd2a0019c5f94c09" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44825 } }
I20260926 20:24:39.809624 18090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:24:39.810860 18097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14e6dedc94be4638bd2a0019c5f94c09. Latest consensus state: current_term: 1 leader_uuid: "14e6dedc94be4638bd2a0019c5f94c09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14e6dedc94be4638bd2a0019c5f94c09" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44825 } } }
I20260926 20:24:39.810981 18097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09 [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:39.811302 18096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14e6dedc94be4638bd2a0019c5f94c09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14e6dedc94be4638bd2a0019c5f94c09" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44825 } } }
I20260926 20:24:39.811398 18096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09 [sys.catalog]: This master's current role is: LEADER
W20260926 20:24:39.814435 18110 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:24:39.814504 18110 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:24:39.814581 18104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:24:39.815291 18104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:24:39.821056 18104 catalog_manager.cc:1347] Generated new cluster ID: b99c4ddf710e4344b2dce75aec47556c
I20260926 20:24:39.821128 18104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:24:39.857275 18104 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:24:39.858232 18104 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:24:39.865015 18104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09: Generated new TSK 0
I20260926 20:24:39.865916 18104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:24:39.880188 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44825
--builtin_ntp_servers=127.0.116.212:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:24:39.995249 18114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:39.995604 18114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:39.995677 18114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:39.999869 18114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:39.999969 18114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:40.000066 18114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:24:40.005020 18114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:40.006202 18114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:40.007699 18114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:40.017763 18119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:40.017833 18122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:40.017845 18120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:40.018030 18114 server_base.cc:1034] running on GCE node
I20260926 20:24:40.018501 18114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:40.019160 18114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:40.020339 18114 hybrid_clock.cc:648] HybridClock initialized: now 1790454280020302 us; error 48 us; skew 500 ppm
I20260926 20:24:40.022123 18114 webserver.cc:460] Webserver started at http://127.0.116.193:35987/ using document root <none> and password file <none>
I20260926 20:24:40.022701 18114 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:40.022799 18114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:40.023025 18114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:40.025092 18114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "ce87d8a1af4041adb44aecda19bad64c"
format_stamp: "Formatted at 2026-09-26 20:24:40 on dist-test-slave-qsm9"
server_key: "fd3a94061784dd5b3f7b9fcc644bf473"
server_key_iv: "26f9904d22038ce8f32e8a06fa563276"
server_key_version: "encryptionkey@0"
I20260926 20:24:40.025674 18114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "ce87d8a1af4041adb44aecda19bad64c"
format_stamp: "Formatted at 2026-09-26 20:24:40 on dist-test-slave-qsm9"
server_key: "fd3a94061784dd5b3f7b9fcc644bf473"
server_key_iv: "26f9904d22038ce8f32e8a06fa563276"
server_key_version: "encryptionkey@0"
I20260926 20:24:40.029580 18114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:24:40.032038 18128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:40.033335 18114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:24:40.033509 18114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "ce87d8a1af4041adb44aecda19bad64c"
format_stamp: "Formatted at 2026-09-26 20:24:40 on dist-test-slave-qsm9"
server_key: "fd3a94061784dd5b3f7b9fcc644bf473"
server_key_iv: "26f9904d22038ce8f32e8a06fa563276"
server_key_version: "encryptionkey@0"
I20260926 20:24:40.033653 18114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:40.051753 18114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:40.052687 18114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:40.052912 18114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:40.053550 18114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:40.054564 18114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:24:40.054638 18114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:40.054730 18114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:24:40.054764 18114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:40.064993 18114 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:43355
I20260926 20:24:40.065039 18241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:43355 every 8 connection(s)
I20260926 20:24:40.065999 18114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:24:40.067286 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 18114
I20260926 20:24:40.067414 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:24:40.067757 467 external_mini_cluster.cc:1428] Setting key d710be2c3daef7711551b5e64e61de59
I20260926 20:24:40.069890 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44825
--builtin_ntp_servers=127.0.116.212:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:24:40.077137 18093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:40.068240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:55625 (local address 127.0.116.254:44825)
0926 20:24:40.068416 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:40.068420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:40.069915 (+ 1495us) server_negotiation.cc:407] Connection header received
0926 20:24:40.071592 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:40.071596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:40.071666 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:40.071769 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:40.072811 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.073412 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.074753 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.075010 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.075131 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:40.076563 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:40.076567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:40.076569 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:40.076583 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:40.076612 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:40.076621 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:40.076712 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:40.076900 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 20:24:40.076973 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:24:40.078099 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:40.068817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44825 (local address 127.0.116.193:55625)
0926 20:24:40.069727 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:40.069787 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:40.071301 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:40.071993 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:40.072002 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:40.072101 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:40.072652 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:40.072666 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.073559 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.073563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:40.074599 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:40.074610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.075196 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.075201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:40.075399 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:40.076216 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:24:40.076218 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:40.076265 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:24:40.076341 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:40.076714 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:40.076728 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:40.076736 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 20:24:40.076882 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 20:24:40.077198 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":181,"threads_started":1}
I20260926 20:24:40.079675 18242 heartbeater.cc:347] Connected to a master server at 127.0.116.254:44825
I20260926 20:24:40.079959 18242 heartbeater.cc:464] Registering TS with master...
I20260926 20:24:40.080570 18242 heartbeater.cc:511] Master 127.0.116.254:44825 requested a full tablet report, sending...
I20260926 20:24:40.082489 18054 ts_manager.cc:194] Registered new tserver with Master: ce87d8a1af4041adb44aecda19bad64c (127.0.116.193:43355)
I20260926 20:24:40.084162 18054 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.116.193:55625
W20260926 20:24:40.188668 18245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:40.188990 18245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:40.189033 18245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:40.193250 18245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:40.193326 18245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:40.193387 18245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:24:40.198382 18245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:40.199460 18245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:40.200922 18245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:40.211313 18251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:40.211297 18250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:40.211289 18253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:40.211536 18245 server_base.cc:1034] running on GCE node
I20260926 20:24:40.212117 18245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:40.212747 18245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:40.213977 18245 hybrid_clock.cc:648] HybridClock initialized: now 1790454280213957 us; error 37 us; skew 500 ppm
I20260926 20:24:40.216135 18245 webserver.cc:460] Webserver started at http://127.0.116.194:35229/ using document root <none> and password file <none>
I20260926 20:24:40.216750 18245 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:40.216851 18245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:40.217098 18245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:40.219237 18245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "180700730d6948e7999366fea588049c"
format_stamp: "Formatted at 2026-09-26 20:24:40 on dist-test-slave-qsm9"
server_key: "c21ea063ecf68628ca16c48a660a11e6"
server_key_iv: "1de646ec6496b05684b6c37a084b4c11"
server_key_version: "encryptionkey@0"
I20260926 20:24:40.219899 18245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "180700730d6948e7999366fea588049c"
format_stamp: "Formatted at 2026-09-26 20:24:40 on dist-test-slave-qsm9"
server_key: "c21ea063ecf68628ca16c48a660a11e6"
server_key_iv: "1de646ec6496b05684b6c37a084b4c11"
server_key_version: "encryptionkey@0"
I20260926 20:24:40.223829 18245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:24:40.226449 18259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:40.227911 18245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:24:40.228057 18245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "180700730d6948e7999366fea588049c"
format_stamp: "Formatted at 2026-09-26 20:24:40 on dist-test-slave-qsm9"
server_key: "c21ea063ecf68628ca16c48a660a11e6"
server_key_iv: "1de646ec6496b05684b6c37a084b4c11"
server_key_version: "encryptionkey@0"
I20260926 20:24:40.228176 18245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:40.266873 18245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:40.267735 18245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:40.267957 18245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:40.268625 18245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:40.269733 18245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:24:40.269788 18245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:40.269865 18245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:24:40.269904 18245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:40.280857 18245 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:42331
I20260926 20:24:40.280885 18372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:42331 every 8 connection(s)
I20260926 20:24:40.281855 18245 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:24:40.287386 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 18245
I20260926 20:24:40.287564 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:24:40.287891 467 external_mini_cluster.cc:1428] Setting key e8348a49c6dcac02e03ceea04c203bcc
I20260926 20:24:40.290254 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44825
--builtin_ntp_servers=127.0.116.212:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:24:40.292626 18093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:40.283958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:57565 (local address 127.0.116.254:44825)
0926 20:24:40.284143 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:40.284148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:40.285284 (+ 1136us) server_negotiation.cc:407] Connection header received
0926 20:24:40.286668 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:40.286671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:40.286766 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:40.287028 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:40.288040 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.288643 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.290172 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.290620 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.290806 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:40.292103 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:40.292106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:40.292108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:40.292121 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:40.292142 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:40.292151 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:40.292220 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:40.292453 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 20:24:40.292513 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 20:24:40.293875 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:40.284352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44825 (local address 127.0.116.194:57565)
0926 20:24:40.285101 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:40.285143 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:40.286447 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:40.287185 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:40.287192 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:40.287282 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:40.287900 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:40.287914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.288803 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.288807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:40.290025 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:40.290035 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.290844 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.290847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:40.290998 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:40.291709 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:24:40.291713 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:40.291782 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:24:40.291935 (+ 153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:40.292243 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:40.292252 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:40.292267 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 20:24:40.292414 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 20:24:40.292758 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":135,"threads_started":1}
I20260926 20:24:40.295248 18373 heartbeater.cc:347] Connected to a master server at 127.0.116.254:44825
I20260926 20:24:40.295598 18373 heartbeater.cc:464] Registering TS with master...
I20260926 20:24:40.296206 18373 heartbeater.cc:511] Master 127.0.116.254:44825 requested a full tablet report, sending...
I20260926 20:24:40.297453 18054 ts_manager.cc:194] Registered new tserver with Master: 180700730d6948e7999366fea588049c (127.0.116.194:42331)
I20260926 20:24:40.298116 18054 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.116.194:57565
W20260926 20:24:40.411307 18376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:40.411652 18376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:40.411731 18376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:40.416402 18376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:40.416463 18376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:40.416527 18376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:24:40.421741 18376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:40.422993 18376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:40.424479 18376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:40.434551 18381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:40.434628 18376 server_base.cc:1034] running on GCE node
W20260926 20:24:40.434553 18382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:40.434576 18384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:40.435196 18376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:40.435978 18376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:40.437161 18376 hybrid_clock.cc:648] HybridClock initialized: now 1790454280437144 us; error 29 us; skew 500 ppm
I20260926 20:24:40.439476 18376 webserver.cc:460] Webserver started at http://127.0.116.195:41875/ using document root <none> and password file <none>
I20260926 20:24:40.440198 18376 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:40.440266 18376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:40.440507 18376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:40.442807 18376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "cc662624d1c64402af192c43727d752d"
format_stamp: "Formatted at 2026-09-26 20:24:40 on dist-test-slave-qsm9"
server_key: "b28391678defe877d583a2d6e2fe9a86"
server_key_iv: "853dded6ba9f10f7d6d9764c020e1782"
server_key_version: "encryptionkey@0"
I20260926 20:24:40.443423 18376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "cc662624d1c64402af192c43727d752d"
format_stamp: "Formatted at 2026-09-26 20:24:40 on dist-test-slave-qsm9"
server_key: "b28391678defe877d583a2d6e2fe9a86"
server_key_iv: "853dded6ba9f10f7d6d9764c020e1782"
server_key_version: "encryptionkey@0"
I20260926 20:24:40.447614 18376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:24:40.450433 18390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:40.451882 18376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:24:40.452013 18376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "cc662624d1c64402af192c43727d752d"
format_stamp: "Formatted at 2026-09-26 20:24:40 on dist-test-slave-qsm9"
server_key: "b28391678defe877d583a2d6e2fe9a86"
server_key_iv: "853dded6ba9f10f7d6d9764c020e1782"
server_key_version: "encryptionkey@0"
I20260926 20:24:40.452132 18376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:40.468503 18376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:40.469236 18376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:40.469450 18376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:40.470068 18376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:40.471063 18376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:24:40.471110 18376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:40.471150 18376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:24:40.471191 18376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:40.480262 18376 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:44787
I20260926 20:24:40.480381 18503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:44787 every 8 connection(s)
I20260926 20:24:40.481282 18376 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:24:40.487922 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 18376
I20260926 20:24:40.488052 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:24:40.488324 467 external_mini_cluster.cc:1428] Setting key 98a9bb4da7c5c25dffa988fcc8d4b0ac
I20260926 20:24:40.491528 18093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:40.483308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:52521 (local address 127.0.116.254:44825)
0926 20:24:40.483466 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:40.483470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:40.484641 (+ 1171us) server_negotiation.cc:407] Connection header received
0926 20:24:40.486099 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:40.486102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:40.486151 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:40.486241 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:40.487229 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.487989 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.489436 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.489732 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.489842 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:40.490981 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:40.490985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:40.490987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:40.491009 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:40.491035 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:40.491043 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:40.491117 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:40.491288 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 20:24:40.491362 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:24:40.492398 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:40.483796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44825 (local address 127.0.116.195:52521)
0926 20:24:40.484466 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:40.484507 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:40.485860 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:40.486464 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:40.486472 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:40.486560 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:40.487064 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:40.487077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.488175 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.488178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:40.489314 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:40.489323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.489869 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.489874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:40.490046 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:40.490717 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:24:40.490719 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:40.490761 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:24:40.490829 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:40.491130 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:40.491141 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:40.491153 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 20:24:40.491277 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 20:24:40.491614 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":111,"threads_started":1}
I20260926 20:24:40.493541 18504 heartbeater.cc:347] Connected to a master server at 127.0.116.254:44825
I20260926 20:24:40.493805 18504 heartbeater.cc:464] Registering TS with master...
I20260926 20:24:40.494356 18504 heartbeater.cc:511] Master 127.0.116.254:44825 requested a full tablet report, sending...
I20260926 20:24:40.495472 18054 ts_manager.cc:194] Registered new tserver with Master: cc662624d1c64402af192c43727d752d (127.0.116.195:44787)
I20260926 20:24:40.496142 18054 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.116.195:52521
I20260926 20:24:40.502846 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:24:40.509271 18093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:40.505153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59820 (local address 127.0.116.254:44825)
0926 20:24:40.505302 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:40.505305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:40.505460 (+ 155us) server_negotiation.cc:407] Connection header received
0926 20:24:40.505545 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:40.505547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:40.505594 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:40.505660 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:40.506045 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.506578 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.507587 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.507810 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.507897 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:40.508733 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:40.508736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:40.508738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:40.508751 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:40.508771 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:40.508779 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:40.508838 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:40.508991 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 20:24:40.509055 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:24:40.512573 18054 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:24:40.515699 18054 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:24:40.531416 18520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:40.526257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49088 (local address 127.0.116.195:44787)
0926 20:24:40.526665 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:40.526673 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:40.526692 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:24:40.526786 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:40.526791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:40.527013 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:40.527162 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:40.527324 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.528495 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.529865 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.530636 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.530773 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:40.530980 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:40.531076 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 20:24:40.531182 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":127,"threads_started":1}
I20260926 20:24:40.531804 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:40.525869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42276 (local address 127.0.116.194:42331)
0926 20:24:40.526318 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:40.526332 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:40.526355 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:24:40.526433 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:40.526441 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:40.526641 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:40.526820 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:40.527093 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.528269 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.530123 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.530643 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.530773 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:40.530977 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:40.531530 (+ 553us) server_negotiation.cc:299] Negotiation successful
0926 20:24:40.531634 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":154,"threads_started":1}
I20260926 20:24:40.531900 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:40.525052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57614 (local address 127.0.116.193:43355)
0926 20:24:40.525943 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:40.525956 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:40.525984 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:24:40.526077 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:40.526082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:40.526314 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:40.526831 (+ 517us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:40.526974 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.528268 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.530076 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.530795 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.531249 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:40.531613 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:40.531668 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 20:24:40.531758 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":102,"threads_started":1}
I20260926 20:24:40.531437 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:40.525789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:44787 (local address 127.0.0.1:49088)
0926 20:24:40.526283 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:40.526297 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:40.526420 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:40.527130 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:40.527132 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:40.527149 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:40.527222 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:40.527227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.528615 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.528619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:40.529731 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:40.529740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.530784 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.530787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:40.530901 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:40.530938 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 20:24:40.531050 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 20:24:40.531238 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":168,"thread_start_us":161,"threads_started":1}
I20260926 20:24:40.532425 18515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:40.524887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:43355 (local address 127.0.0.1:57614)
0926 20:24:40.525415 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:40.525466 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:40.525597 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:40.526502 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:40.526508 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:40.526532 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:40.526681 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:40.526690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.528391 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.528394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:40.529944 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:40.529956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.530926 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.530930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:40.531033 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:40.531060 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 20:24:40.531109 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 20:24:40.531934 (+ 825us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":728,"thread_start_us":94,"threads_started":1}
I20260926 20:24:40.532425 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:40.525088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:42331 (local address 127.0.0.1:42276)
0926 20:24:40.525594 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:40.525606 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:40.525682 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:40.526797 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:40.526800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:40.526818 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:40.526927 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:40.526933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.528431 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.528434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:40.529984 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:40.529995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.531198 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.531202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:40.531277 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:40.531297 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 20:24:40.531364 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 20:24:40.531421 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":279,"thread_start_us":89,"threads_started":1}
I20260926 20:24:40.535200 18307 tablet_service.cc:1467] Processing CreateTablet for tablet 60314c48168a4b4dbf2f65b5ab0c602b (DEFAULT_TABLE table=test-table [id=b63cde21d02d4a81bd8938a6d61a6750]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:24:40.535360 18176 tablet_service.cc:1467] Processing CreateTablet for tablet 60314c48168a4b4dbf2f65b5ab0c602b (DEFAULT_TABLE table=test-table [id=b63cde21d02d4a81bd8938a6d61a6750]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:24:40.535360 18438 tablet_service.cc:1467] Processing CreateTablet for tablet 60314c48168a4b4dbf2f65b5ab0c602b (DEFAULT_TABLE table=test-table [id=b63cde21d02d4a81bd8938a6d61a6750]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:24:40.536391 18307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60314c48168a4b4dbf2f65b5ab0c602b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:40.536398 18438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60314c48168a4b4dbf2f65b5ab0c602b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:40.536408 18176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60314c48168a4b4dbf2f65b5ab0c602b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:40.543010 18521 tablet_bootstrap.cc:492] T 60314c48168a4b4dbf2f65b5ab0c602b P 180700730d6948e7999366fea588049c: Bootstrap starting.
I20260926 20:24:40.544639 18522 tablet_bootstrap.cc:492] T 60314c48168a4b4dbf2f65b5ab0c602b P ce87d8a1af4041adb44aecda19bad64c: Bootstrap starting.
I20260926 20:24:40.544639 18523 tablet_bootstrap.cc:492] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d: Bootstrap starting.
I20260926 20:24:40.545315 18521 tablet_bootstrap.cc:654] T 60314c48168a4b4dbf2f65b5ab0c602b P 180700730d6948e7999366fea588049c: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:40.546085 18521 log.cc:824] T 60314c48168a4b4dbf2f65b5ab0c602b P 180700730d6948e7999366fea588049c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:40.546924 18523 tablet_bootstrap.cc:654] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:40.546921 18522 tablet_bootstrap.cc:654] T 60314c48168a4b4dbf2f65b5ab0c602b P ce87d8a1af4041adb44aecda19bad64c: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:40.547845 18522 log.cc:824] T 60314c48168a4b4dbf2f65b5ab0c602b P ce87d8a1af4041adb44aecda19bad64c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:40.547845 18523 log.cc:824] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:40.548724 18521 tablet_bootstrap.cc:492] T 60314c48168a4b4dbf2f65b5ab0c602b P 180700730d6948e7999366fea588049c: No bootstrap required, opened a new log
I20260926 20:24:40.548945 18521 ts_tablet_manager.cc:1397] T 60314c48168a4b4dbf2f65b5ab0c602b P 180700730d6948e7999366fea588049c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:24:40.550010 18523 tablet_bootstrap.cc:492] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d: No bootstrap required, opened a new log
I20260926 20:24:40.550010 18522 tablet_bootstrap.cc:492] T 60314c48168a4b4dbf2f65b5ab0c602b P ce87d8a1af4041adb44aecda19bad64c: No bootstrap required, opened a new log
I20260926 20:24:40.550194 18523 ts_tablet_manager.cc:1397] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:24:40.550194 18522 ts_tablet_manager.cc:1397] T 60314c48168a4b4dbf2f65b5ab0c602b P ce87d8a1af4041adb44aecda19bad64c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:24:40.552460 18521 raft_consensus.cc:348] T 60314c48168a4b4dbf2f65b5ab0c602b P 180700730d6948e7999366fea588049c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce87d8a1af4041adb44aecda19bad64c" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43355 } } peers { permanent_uuid: "180700730d6948e7999366fea588049c" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42331 } } peers { permanent_uuid: "cc662624d1c64402af192c43727d752d" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 44787 } }
I20260926 20:24:40.552711 18521 raft_consensus.cc:374] T 60314c48168a4b4dbf2f65b5ab0c602b P 180700730d6948e7999366fea588049c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:40.552770 18521 raft_consensus.cc:729] T 60314c48168a4b4dbf2f65b5ab0c602b P 180700730d6948e7999366fea588049c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 180700730d6948e7999366fea588049c, State: Initialized, Role: FOLLOWER
I20260926 20:24:40.553226 18521 consensus_queue.cc:260] T 60314c48168a4b4dbf2f65b5ab0c602b P 180700730d6948e7999366fea588049c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce87d8a1af4041adb44aecda19bad64c" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43355 } } peers { permanent_uuid: "180700730d6948e7999366fea588049c" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42331 } } peers { permanent_uuid: "cc662624d1c64402af192c43727d752d" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 44787 } }
I20260926 20:24:40.554318 18521 ts_tablet_manager.cc:1428] T 60314c48168a4b4dbf2f65b5ab0c602b P 180700730d6948e7999366fea588049c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 20:24:40.554684 18373 heartbeater.cc:503] Master 127.0.116.254:44825 was elected leader, sending a full tablet report...
I20260926 20:24:40.554761 18523 raft_consensus.cc:348] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce87d8a1af4041adb44aecda19bad64c" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43355 } } peers { permanent_uuid: "180700730d6948e7999366fea588049c" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42331 } } peers { permanent_uuid: "cc662624d1c64402af192c43727d752d" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 44787 } }
I20260926 20:24:40.554761 18522 raft_consensus.cc:348] T 60314c48168a4b4dbf2f65b5ab0c602b P ce87d8a1af4041adb44aecda19bad64c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce87d8a1af4041adb44aecda19bad64c" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43355 } } peers { permanent_uuid: "180700730d6948e7999366fea588049c" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42331 } } peers { permanent_uuid: "cc662624d1c64402af192c43727d752d" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 44787 } }
I20260926 20:24:40.555054 18522 raft_consensus.cc:374] T 60314c48168a4b4dbf2f65b5ab0c602b P ce87d8a1af4041adb44aecda19bad64c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:40.555053 18523 raft_consensus.cc:374] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:40.555114 18523 raft_consensus.cc:729] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc662624d1c64402af192c43727d752d, State: Initialized, Role: FOLLOWER
I20260926 20:24:40.555114 18522 raft_consensus.cc:729] T 60314c48168a4b4dbf2f65b5ab0c602b P ce87d8a1af4041adb44aecda19bad64c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce87d8a1af4041adb44aecda19bad64c, State: Initialized, Role: FOLLOWER
I20260926 20:24:40.555622 18522 consensus_queue.cc:260] T 60314c48168a4b4dbf2f65b5ab0c602b P ce87d8a1af4041adb44aecda19bad64c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce87d8a1af4041adb44aecda19bad64c" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43355 } } peers { permanent_uuid: "180700730d6948e7999366fea588049c" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42331 } } peers { permanent_uuid: "cc662624d1c64402af192c43727d752d" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 44787 } }
I20260926 20:24:40.555622 18523 consensus_queue.cc:260] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce87d8a1af4041adb44aecda19bad64c" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43355 } } peers { permanent_uuid: "180700730d6948e7999366fea588049c" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42331 } } peers { permanent_uuid: "cc662624d1c64402af192c43727d752d" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 44787 } }
I20260926 20:24:40.556718 18523 ts_tablet_manager.cc:1428] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260926 20:24:40.556718 18522 ts_tablet_manager.cc:1428] T 60314c48168a4b4dbf2f65b5ab0c602b P ce87d8a1af4041adb44aecda19bad64c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 20:24:40.557104 18504 heartbeater.cc:503] Master 127.0.116.254:44825 was elected leader, sending a full tablet report...
I20260926 20:24:40.558008 18242 heartbeater.cc:503] Master 127.0.116.254:44825 was elected leader, sending a full tablet report...
W20260926 20:24:40.567216 18243 tablet.cc:2369] T 60314c48168a4b4dbf2f65b5ab0c602b P ce87d8a1af4041adb44aecda19bad64c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:24:40.732340 18505 tablet.cc:2369] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:24:40.782550 18529 raft_consensus.cc:482] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:24:40.782780 18529 raft_consensus.cc:504] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce87d8a1af4041adb44aecda19bad64c" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43355 } } peers { permanent_uuid: "180700730d6948e7999366fea588049c" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42331 } } peers { permanent_uuid: "cc662624d1c64402af192c43727d752d" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 44787 } }
W20260926 20:24:40.783087 18374 tablet.cc:2369] T 60314c48168a4b4dbf2f65b5ab0c602b P 180700730d6948e7999366fea588049c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:24:40.784296 18529 leader_election.cc:288] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce87d8a1af4041adb44aecda19bad64c (127.0.116.193:43355), 180700730d6948e7999366fea588049c (127.0.116.194:42331)
I20260926 20:24:40.788656 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:40.784518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:43355 (local address 127.0.116.195:44545)
0926 20:24:40.784751 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:40.784767 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:40.784886 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:40.785218 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:40.785221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:40.785238 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:40.785368 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:40.785375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.786469 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.786472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:40.787370 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:40.787379 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.788252 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.788254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:40.788328 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:40.788344 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 20:24:40.788438 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 20:24:40.788516 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":51}
I20260926 20:24:40.788656 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:40.784738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:44545 (local address 127.0.116.193:43355)
0926 20:24:40.784868 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:40.784873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:40.784892 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:24:40.785042 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:40.785046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:40.785112 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:40.785205 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:40.785512 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.786333 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.787543 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.788019 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.788156 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:40.788199 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:40.788456 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 20:24:40.788532 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:24:40.789383 18196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60314c48168a4b4dbf2f65b5ab0c602b" candidate_uuid: "cc662624d1c64402af192c43727d752d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce87d8a1af4041adb44aecda19bad64c" is_pre_election: true
I20260926 20:24:40.789467 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:40.784627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:48411 (local address 127.0.116.194:42331)
0926 20:24:40.784754 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:40.784762 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:40.785289 (+ 527us) server_negotiation.cc:407] Connection header received
0926 20:24:40.785494 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:40.785498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:40.785551 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:40.785636 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:40.785953 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.786473 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.788075 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.788748 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.788878 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:40.788928 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:40.789257 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 20:24:40.789337 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:24:40.789729 18196 raft_consensus.cc:2454] T 60314c48168a4b4dbf2f65b5ab0c602b P ce87d8a1af4041adb44aecda19bad64c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc662624d1c64402af192c43727d752d in term 0.
I20260926 20:24:40.789824 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:40.784518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:42331 (local address 127.0.116.195:48411)
0926 20:24:40.785170 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:40.785185 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:40.785362 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:40.785647 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:40.785650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:40.785666 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:40.785789 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:40.785794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.786608 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.786611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:40.787862 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:40.787873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.789010 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.789014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:40.789111 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:40.789135 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:24:40.789594 (+ 459us) client_negotiation.cc:240] Negotiation successful
0926 20:24:40.789669 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":258,"thread_start_us":134,"threads_started":1}
I20260926 20:24:40.790267 18392 leader_election.cc:302] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc662624d1c64402af192c43727d752d, ce87d8a1af4041adb44aecda19bad64c; no voters:
I20260926 20:24:40.790586 18529 raft_consensus.cc:2781] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:24:40.790673 18529 raft_consensus.cc:482] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:24:40.790717 18529 raft_consensus.cc:3037] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:40.790879 18327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60314c48168a4b4dbf2f65b5ab0c602b" candidate_uuid: "cc662624d1c64402af192c43727d752d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "180700730d6948e7999366fea588049c" is_pre_election: true
I20260926 20:24:40.791180 18327 raft_consensus.cc:2454] T 60314c48168a4b4dbf2f65b5ab0c602b P 180700730d6948e7999366fea588049c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc662624d1c64402af192c43727d752d in term 0.
I20260926 20:24:40.792081 18529 raft_consensus.cc:504] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce87d8a1af4041adb44aecda19bad64c" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43355 } } peers { permanent_uuid: "180700730d6948e7999366fea588049c" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42331 } } peers { permanent_uuid: "cc662624d1c64402af192c43727d752d" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 44787 } }
I20260926 20:24:40.792492 18529 leader_election.cc:288] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d [CANDIDATE]: Term 1 election: Requested vote from peers ce87d8a1af4041adb44aecda19bad64c (127.0.116.193:43355), 180700730d6948e7999366fea588049c (127.0.116.194:42331)
I20260926 20:24:40.792871 18196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60314c48168a4b4dbf2f65b5ab0c602b" candidate_uuid: "cc662624d1c64402af192c43727d752d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce87d8a1af4041adb44aecda19bad64c"
I20260926 20:24:40.793009 18327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60314c48168a4b4dbf2f65b5ab0c602b" candidate_uuid: "cc662624d1c64402af192c43727d752d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "180700730d6948e7999366fea588049c"
I20260926 20:24:40.793018 18196 raft_consensus.cc:3037] T 60314c48168a4b4dbf2f65b5ab0c602b P ce87d8a1af4041adb44aecda19bad64c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:40.793145 18327 raft_consensus.cc:3037] T 60314c48168a4b4dbf2f65b5ab0c602b P 180700730d6948e7999366fea588049c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:40.794373 18327 raft_consensus.cc:2454] T 60314c48168a4b4dbf2f65b5ab0c602b P 180700730d6948e7999366fea588049c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc662624d1c64402af192c43727d752d in term 1.
I20260926 20:24:40.794399 18196 raft_consensus.cc:2454] T 60314c48168a4b4dbf2f65b5ab0c602b P ce87d8a1af4041adb44aecda19bad64c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc662624d1c64402af192c43727d752d in term 1.
I20260926 20:24:40.794744 18394 leader_election.cc:302] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 180700730d6948e7999366fea588049c, cc662624d1c64402af192c43727d752d; no voters:
I20260926 20:24:40.794975 18529 raft_consensus.cc:2781] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:24:40.795198 18529 raft_consensus.cc:686] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d [term 1 LEADER]: Becoming Leader. State: Replica: cc662624d1c64402af192c43727d752d, State: Running, Role: LEADER
I20260926 20:24:40.795575 18529 consensus_queue.cc:237] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce87d8a1af4041adb44aecda19bad64c" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43355 } } peers { permanent_uuid: "180700730d6948e7999366fea588049c" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42331 } } peers { permanent_uuid: "cc662624d1c64402af192c43727d752d" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 44787 } }
I20260926 20:24:40.799113 18054 catalog_manager.cc:5495] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d reported cstate change: term changed from 0 to 1, leader changed from <none> to cc662624d1c64402af192c43727d752d (127.0.116.195). New cstate: current_term: 1 leader_uuid: "cc662624d1c64402af192c43727d752d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce87d8a1af4041adb44aecda19bad64c" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "180700730d6948e7999366fea588049c" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc662624d1c64402af192c43727d752d" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 44787 } health_report { overall_health: HEALTHY } } }
I20260926 20:24:40.824730 18520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:40.821137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49092 (local address 127.0.116.195:44787)
0926 20:24:40.821279 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:40.821290 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:40.821306 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:24:40.821399 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:40.821401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:40.821456 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:40.821543 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:24:40.821793 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.822291 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.823236 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:40.823448 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:40.823721 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:40.823993 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:24:40.824412 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:24:40.824504 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:40.824576 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 20:24:40.824621 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:24:40.830684 18327 raft_consensus.cc:1261] T 60314c48168a4b4dbf2f65b5ab0c602b P 180700730d6948e7999366fea588049c [term 1 FOLLOWER]: Refusing update from remote peer cc662624d1c64402af192c43727d752d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:24:40.830685 18196 raft_consensus.cc:1261] T 60314c48168a4b4dbf2f65b5ab0c602b P ce87d8a1af4041adb44aecda19bad64c [term 1 FOLLOWER]: Refusing update from remote peer cc662624d1c64402af192c43727d752d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:24:40.831475 18531 consensus_queue.cc:1035] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d [LEADER]: Connected to new peer: Peer: permanent_uuid: "180700730d6948e7999366fea588049c" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 42331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:24:40.831714 18529 consensus_queue.cc:1035] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce87d8a1af4041adb44aecda19bad64c" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:24:40.840492 18536 mvcc.cc:205] Tried to move back new op lower bound from 7333700734275198976 to 7333700734141497344. Current Snapshot: MvccSnapshot[applied={T|T < 7333700734275198976}]
I20260926 20:24:40.841301 18535 mvcc.cc:205] Tried to move back new op lower bound from 7333700734275198976 to 7333700734141497344. Current Snapshot: MvccSnapshot[applied={T|T < 7333700734275198976}]
I20260926 20:24:40.846647 18054 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:24:40.846871 18054 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:24:40.849696 18054 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14e6dedc94be4638bd2a0019c5f94c09: Sending DeleteTablet for 3 replicas of tablet 60314c48168a4b4dbf2f65b5ab0c602b
I20260926 20:24:40.850554 18176 tablet_service.cc:1514] Processing DeleteTablet for tablet 60314c48168a4b4dbf2f65b5ab0c602b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:24:40 UTC) from {username='slave'} at 127.0.0.1:57614
I20260926 20:24:40.850669 18307 tablet_service.cc:1514] Processing DeleteTablet for tablet 60314c48168a4b4dbf2f65b5ab0c602b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:24:40 UTC) from {username='slave'} at 127.0.0.1:42276
I20260926 20:24:40.851181 18543 tablet_replica.cc:330] T 60314c48168a4b4dbf2f65b5ab0c602b P ce87d8a1af4041adb44aecda19bad64c: stopping tablet replica
I20260926 20:24:40.851293 18544 tablet_replica.cc:330] T 60314c48168a4b4dbf2f65b5ab0c602b P 180700730d6948e7999366fea588049c: stopping tablet replica
I20260926 20:24:40.851274 18438 tablet_service.cc:1514] Processing DeleteTablet for tablet 60314c48168a4b4dbf2f65b5ab0c602b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:24:40 UTC) from {username='slave'} at 127.0.0.1:49088
I20260926 20:24:40.851596 18543 raft_consensus.cc:2229] T 60314c48168a4b4dbf2f65b5ab0c602b P ce87d8a1af4041adb44aecda19bad64c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:24:40.851656 18544 raft_consensus.cc:2229] T 60314c48168a4b4dbf2f65b5ab0c602b P 180700730d6948e7999366fea588049c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:24:40.851946 18543 raft_consensus.cc:2258] T 60314c48168a4b4dbf2f65b5ab0c602b P ce87d8a1af4041adb44aecda19bad64c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:24:40.851972 18544 raft_consensus.cc:2258] T 60314c48168a4b4dbf2f65b5ab0c602b P 180700730d6948e7999366fea588049c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:24:40.852564 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 18114
I20260926 20:24:40.854384 18544 ts_tablet_manager.cc:1905] T 60314c48168a4b4dbf2f65b5ab0c602b P 180700730d6948e7999366fea588049c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:24:40.857882 18544 ts_tablet_manager.cc:1918] T 60314c48168a4b4dbf2f65b5ab0c602b P 180700730d6948e7999366fea588049c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:24:40.857993 18544 log.cc:1196] T 60314c48168a4b4dbf2f65b5ab0c602b P 180700730d6948e7999366fea588049c: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/60314c48168a4b4dbf2f65b5ab0c602b
I20260926 20:24:40.858309 18544 ts_tablet_manager.cc:1939] T 60314c48168a4b4dbf2f65b5ab0c602b P 180700730d6948e7999366fea588049c: Deleting consensus metadata
I20260926 20:24:40.859741 18545 tablet_replica.cc:330] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d: stopping tablet replica
I20260926 20:24:40.860037 18041 catalog_manager.cc:4841] TS 180700730d6948e7999366fea588049c (127.0.116.194:42331): tablet 60314c48168a4b4dbf2f65b5ab0c602b (table test-table [id=b63cde21d02d4a81bd8938a6d61a6750]) successfully deleted
I20260926 20:24:40.860229 18545 raft_consensus.cc:2229] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:24:40.860639 18545 raft_consensus.cc:2258] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:24:40.862061 18545 ts_tablet_manager.cc:1905] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 20:24:40.862090 18039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.116.193:43355 (error 108)
I20260926 20:24:40.863125 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:40.862814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:43355 (local address 127.0.0.1:57618)
0926 20:24:40.862945 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:40.863022 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.193:43355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 20:24:40.863188 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 18245
W20260926 20:24:40.863837 18039 catalog_manager.cc:4584] TS ce87d8a1af4041adb44aecda19bad64c (127.0.116.193:43355): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60314c48168a4b4dbf2f65b5ab0c602b: Network error: Client connection negotiation failed: client connection to 127.0.116.193:43355: connect: Connection refused (error 111)
I20260926 20:24:40.864840 18545 ts_tablet_manager.cc:1918] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:24:40.864943 18545 log.cc:1196] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/60314c48168a4b4dbf2f65b5ab0c602b
I20260926 20:24:40.865307 18545 ts_tablet_manager.cc:1939] T 60314c48168a4b4dbf2f65b5ab0c602b P cc662624d1c64402af192c43727d752d: Deleting consensus metadata
I20260926 20:24:40.866633 18038 catalog_manager.cc:4841] TS cc662624d1c64402af192c43727d752d (127.0.116.195:44787): tablet 60314c48168a4b4dbf2f65b5ab0c602b (table test-table [id=b63cde21d02d4a81bd8938a6d61a6750]) successfully deleted
I20260926 20:24:40.870623 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 18376
I20260926 20:24:40.876363 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 18023
2026-09-26T20:24:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1305 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:24:40 dist-test-slave-qsm9 krb5kdc[18549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:24:40 dist-test-slave-qsm9 krb5kdc[18549](info): set up 2 sockets
Sep 26 20:24:40 dist-test-slave-qsm9 krb5kdc[18549](info): commencing operation
krb5kdc: starting...
W20260926 20:24:41.947979 467 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:24:42 dist-test-slave-qsm9 krb5kdc[18549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:24:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:24:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:24:42.075404 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:40713
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:40713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:24:42.192312 18562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:42.192615 18562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:42.192658 18562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:42.197280 18562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:24:42.197355 18562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:42.197379 18562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:42.197397 18562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:24:42.197413 18562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:24:42.202960 18562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:40713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:40713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:42.204102 18562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:42.205111 18562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:42.214087 18568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:42.214130 18562 server_base.cc:1034] running on GCE node
W20260926 20:24:42.214062 18567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:42.214072 18570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:42.214902 18562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:42.216104 18562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:42.217306 18562 hybrid_clock.cc:648] HybridClock initialized: now 1790454282217271 us; error 50 us; skew 500 ppm
Sep 26 20:24:42 dist-test-slave-qsm9 krb5kdc[18549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:42.225104 18562 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:24:42.226158 18562 webserver.cc:460] Webserver started at http://127.0.116.254:34249/ using document root <none> and password file <none>
I20260926 20:24:42.226774 18562 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:42.226872 18562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:42.227121 18562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:42.229373 18562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "721942f8ae9e4dd1a29ce024011b6340"
format_stamp: "Formatted at 2026-09-26 20:24:42 on dist-test-slave-qsm9"
server_key: "a33c9430bcfa9bd11e9fcc82ea2dcb01"
server_key_iv: "b5d1b5f581d3227697f20fd35ddcba87"
server_key_version: "encryptionkey@0"
I20260926 20:24:42.229986 18562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "721942f8ae9e4dd1a29ce024011b6340"
format_stamp: "Formatted at 2026-09-26 20:24:42 on dist-test-slave-qsm9"
server_key: "a33c9430bcfa9bd11e9fcc82ea2dcb01"
server_key_iv: "b5d1b5f581d3227697f20fd35ddcba87"
server_key_version: "encryptionkey@0"
I20260926 20:24:42.234172 18562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:24:42.236862 18577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:42.238296 18562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:24:42.238471 18562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "721942f8ae9e4dd1a29ce024011b6340"
format_stamp: "Formatted at 2026-09-26 20:24:42 on dist-test-slave-qsm9"
server_key: "a33c9430bcfa9bd11e9fcc82ea2dcb01"
server_key_iv: "b5d1b5f581d3227697f20fd35ddcba87"
server_key_version: "encryptionkey@0"
I20260926 20:24:42.238613 18562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:42.262919 18562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:42.266280 18562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:42.266506 18562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:42.274376 18562 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:40713
I20260926 20:24:42.274397 18629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:40713 every 8 connection(s)
I20260926 20:24:42.275435 18562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:24:42.278388 18630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:42.282796 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 18562
I20260926 20:24:42.282974 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:24:42.283277 467 external_mini_cluster.cc:1428] Setting key 8916be1a96d0b1fb34b5e6a8c007e12b
I20260926 20:24:42.284765 18630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340: Bootstrap starting.
I20260926 20:24:42.287683 18630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:42.288543 18630 log.cc:824] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:42.290796 18630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340: No bootstrap required, opened a new log
Sep 26 20:24:42 dist-test-slave-qsm9 krb5kdc[18549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:24:42.295451 18630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721942f8ae9e4dd1a29ce024011b6340" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 40713 } }
I20260926 20:24:42.295804 18630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:42.295862 18630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721942f8ae9e4dd1a29ce024011b6340, State: Initialized, Role: FOLLOWER
I20260926 20:24:42.296430 18630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721942f8ae9e4dd1a29ce024011b6340" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 40713 } }
I20260926 20:24:42.296605 18630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:24:42.296679 18630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:24:42.296779 18630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:42.298054 18630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721942f8ae9e4dd1a29ce024011b6340" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 40713 } }
I20260926 20:24:42.298424 18630 leader_election.cc:302] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 721942f8ae9e4dd1a29ce024011b6340; no voters:
I20260926 20:24:42.298779 18630 leader_election.cc:288] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:24:42.298981 18635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:24:42.299216 18635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340 [term 1 LEADER]: Becoming Leader. State: Replica: 721942f8ae9e4dd1a29ce024011b6340, State: Running, Role: LEADER
I20260926 20:24:42.299616 18635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721942f8ae9e4dd1a29ce024011b6340" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 40713 } }
I20260926 20:24:42.300274 18630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:24:42.300595 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:42.285003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.0.116.254:40713)
0926 20:24:42.285482 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:42.285502 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:42.285545 (+ 43us) server_negotiation.cc:407] Connection header received
0926 20:24:42.286475 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:42.286500 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:42.286939 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:42.287331 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:42.288262 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:42.289634 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:42.290572 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:42.291257 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:42.291434 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:42.294579 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:42.294586 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:42.294607 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:42.294642 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:42.296826 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:42.297329 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:42.297335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:42.297341 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:42.297517 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:42.297843 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:42.297846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:42.297848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:42.298396 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:42.298599 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:42.299140 (+ 541us) server_negotiation.cc:299] Negotiation successful
0926 20:24:42.299555 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":147,"threads_started":1}
I20260926 20:24:42.301031 18636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 721942f8ae9e4dd1a29ce024011b6340. Latest consensus state: current_term: 1 leader_uuid: "721942f8ae9e4dd1a29ce024011b6340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721942f8ae9e4dd1a29ce024011b6340" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 40713 } } }
I20260926 20:24:42.301154 18636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340 [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:42.301245 18637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "721942f8ae9e4dd1a29ce024011b6340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721942f8ae9e4dd1a29ce024011b6340" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 40713 } } }
I20260926 20:24:42.301355 18637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340 [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:42.301519 18639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:24:42.304703 18639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:24:42.310981 18639 catalog_manager.cc:1347] Generated new cluster ID: 894332d0daea4b638d04e3b05999bdb4
I20260926 20:24:42.311064 18639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:24:42.338415 18639 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:24:42.339540 18639 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:24:42.345584 18639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340: Generated new TSK 0
I20260926 20:24:42.346414 18639 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
I20260926 20:24:42.384298 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:40713
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:24:42.503018 18656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:42.503326 18656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:42.503374 18656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:42.508049 18656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:42.508131 18656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:42.508199 18656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:24:42.514163 18656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:40713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:42.515302 18656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:42.516453 18656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:42.526347 18661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:42.526328 18664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:42.526347 18662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:42.526422 18656 server_base.cc:1034] running on GCE node
I20260926 20:24:42.526969 18656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:42.527782 18656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:42.529009 18656 hybrid_clock.cc:648] HybridClock initialized: now 1790454282528988 us; error 29 us; skew 500 ppm
Sep 26 20:24:42 dist-test-slave-qsm9 krb5kdc[18549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:42.531853 18656 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:24:42.532740 18656 webserver.cc:460] Webserver started at http://127.0.116.193:34613/ using document root <none> and password file <none>
I20260926 20:24:42.533331 18656 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:42.533427 18656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:42.533665 18656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:42.535802 18656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "d1063cbcdd614eacbe780d20e3d97264"
format_stamp: "Formatted at 2026-09-26 20:24:42 on dist-test-slave-qsm9"
server_key: "ca9536e363c47c372e86f76c23d59bb8"
server_key_iv: "407aaafe5f14cde3e4295fdc24ce4c72"
server_key_version: "encryptionkey@0"
I20260926 20:24:42.536382 18656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "d1063cbcdd614eacbe780d20e3d97264"
format_stamp: "Formatted at 2026-09-26 20:24:42 on dist-test-slave-qsm9"
server_key: "ca9536e363c47c372e86f76c23d59bb8"
server_key_iv: "407aaafe5f14cde3e4295fdc24ce4c72"
server_key_version: "encryptionkey@0"
I20260926 20:24:42.540333 18656 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:24:42.542989 18671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:42.544382 18656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:24:42.544524 18656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "d1063cbcdd614eacbe780d20e3d97264"
format_stamp: "Formatted at 2026-09-26 20:24:42 on dist-test-slave-qsm9"
server_key: "ca9536e363c47c372e86f76c23d59bb8"
server_key_iv: "407aaafe5f14cde3e4295fdc24ce4c72"
server_key_version: "encryptionkey@0"
I20260926 20:24:42.544646 18656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:42.563889 18656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:42.567181 18656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:42.567396 18656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:42.568065 18656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:42.568910 18656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:24:42.568984 18656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:42.569054 18656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:24:42.569105 18656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:42.578459 18656 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:43149
I20260926 20:24:42.578562 18784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:43149 every 8 connection(s)
I20260926 20:24:42.579514 18656 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:24:42.581488 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 18656
I20260926 20:24:42.581629 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:24:42.581939 467 external_mini_cluster.cc:1428] Setting key e0bf1cc949ee561d04acdd4609ffb192
Sep 26 20:24:42 dist-test-slave-qsm9 krb5kdc[18549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.116.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.194@KRBTEST.COM" created.
I20260926 20:24:42.595646 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:42.581576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:40113 (local address 127.0.116.254:40713)
0926 20:24:42.581736 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:42.581742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:42.582884 (+ 1142us) server_negotiation.cc:407] Connection header received
0926 20:24:42.584536 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:42.584540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:42.584597 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:42.584682 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:42.585963 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:42.586525 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:42.587900 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:42.588155 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:42.588305 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:42.591444 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:42.591448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:42.591450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:42.591477 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:42.593179 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:42.593760 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:42.593763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:42.593765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:42.593852 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:42.594299 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:42.594302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:42.594304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:42.594515 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:42.594629 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:42.595287 (+ 658us) server_negotiation.cc:299] Negotiation successful
0926 20:24:42.595437 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:24:42.596586 18787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:42.582082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:40713 (local address 127.0.116.193:40113)
0926 20:24:42.582685 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:42.582734 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:42.584307 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:42.584884 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:42.584891 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:42.585283 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:42.585796 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:42.585810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:42.586671 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:42.586675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:42.587748 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:42.587767 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:42.588346 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:42.588351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:42.588554 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:42.589315 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:24:42.589333 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:42.591208 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:42.593313 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:42.593321 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:42.593340 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:42.593642 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:42.593951 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:42.593957 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:42.593960 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:42.594193 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:42.594629 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:42.594638 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:42.594974 (+ 336us) client_negotiation.cc:769] Sending connection context
0926 20:24:42.595237 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 20:24:42.595629 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":136,"threads_started":1}
I20260926 20:24:42.598091 18785 heartbeater.cc:347] Connected to a master server at 127.0.116.254:40713
I20260926 20:24:42.598349 18785 heartbeater.cc:464] Registering TS with master...
I20260926 20:24:42.598896 18785 heartbeater.cc:511] Master 127.0.116.254:40713 requested a full tablet report, sending...
I20260926 20:24:42.600595 18594 ts_manager.cc:194] Registered new tserver with Master: d1063cbcdd614eacbe780d20e3d97264 (127.0.116.193:43149)
I20260926 20:24:42.601939 18594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:40113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
I20260926 20:24:42.615761 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:40713
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:24:42.729454 18790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:42.729725 18790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:42.729770 18790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:42.733911 18790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:42.733983 18790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:42.734045 18790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:24:42.739665 18790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:40713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:42.740782 18790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:42.741858 18790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:42.752199 18798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:42.752254 18790 server_base.cc:1034] running on GCE node
W20260926 20:24:42.752204 18796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:42.752199 18795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:42.752905 18790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:42.753576 18790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:42.754776 18790 hybrid_clock.cc:648] HybridClock initialized: now 1790454282754753 us; error 34 us; skew 500 ppm
Sep 26 20:24:42 dist-test-slave-qsm9 krb5kdc[18549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:42.757974 18790 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:24:42.758937 18790 webserver.cc:460] Webserver started at http://127.0.116.194:46503/ using document root <none> and password file <none>
I20260926 20:24:42.759603 18790 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:42.759675 18790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:42.759918 18790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:42.762012 18790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "e3e2e94c5ec84483ab1a7b48149d055f"
format_stamp: "Formatted at 2026-09-26 20:24:42 on dist-test-slave-qsm9"
server_key: "c6739c195eb5088dc1919ad57fa8d111"
server_key_iv: "5c60b4c553951146abec6c2865d63770"
server_key_version: "encryptionkey@0"
I20260926 20:24:42.762593 18790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "e3e2e94c5ec84483ab1a7b48149d055f"
format_stamp: "Formatted at 2026-09-26 20:24:42 on dist-test-slave-qsm9"
server_key: "c6739c195eb5088dc1919ad57fa8d111"
server_key_iv: "5c60b4c553951146abec6c2865d63770"
server_key_version: "encryptionkey@0"
I20260926 20:24:42.766511 18790 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 20:24:42.769287 18805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:42.770737 18790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:24:42.770898 18790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "e3e2e94c5ec84483ab1a7b48149d055f"
format_stamp: "Formatted at 2026-09-26 20:24:42 on dist-test-slave-qsm9"
server_key: "c6739c195eb5088dc1919ad57fa8d111"
server_key_iv: "5c60b4c553951146abec6c2865d63770"
server_key_version: "encryptionkey@0"
I20260926 20:24:42.771039 18790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:42.783205 18790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:42.786330 18790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:42.786532 18790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:42.787132 18790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:42.788075 18790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:24:42.788125 18790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:42.788192 18790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:24:42.788228 18790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:42.797746 18790 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:46879
I20260926 20:24:42.797813 18918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:46879 every 8 connection(s)
I20260926 20:24:42.798722 18790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:24:42.802603 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 18790
I20260926 20:24:42.802724 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:24:42.803093 467 external_mini_cluster.cc:1428] Setting key ec59b633749f22a7ebbbb0ff5582fb3b
Sep 26 20:24:42 dist-test-slave-qsm9 krb5kdc[18549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.116.195@KRBTEST.COM; defaulting to no policy
I20260926 20:24:42.813724 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:42.800631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:53511 (local address 127.0.116.254:40713)
0926 20:24:42.800777 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:42.800781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:42.801871 (+ 1090us) server_negotiation.cc:407] Connection header received
0926 20:24:42.803599 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:42.803604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:42.803673 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:42.803764 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:42.805075 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:42.805594 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:42.806719 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:42.806937 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:42.807046 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:42.809959 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:42.809962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:42.809964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:42.809987 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:42.811653 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:42.812172 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:42.812177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:42.812179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:42.812259 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:42.812594 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:42.812598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:42.812600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:42.812808 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:42.812914 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:42.813401 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 20:24:42.813512 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:24:42.814489 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:42.801100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:40713 (local address 127.0.116.194:53511)
0926 20:24:42.801709 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:42.801773 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:42.803313 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:42.803969 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:42.803978 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:42.804404 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:42.804928 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:42.804943 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:42.805718 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:42.805722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:42.806594 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:42.806603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:42.807083 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:42.807087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:42.807311 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:42.808047 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:24:42.808072 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:42.809765 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:42.811779 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:42.811784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:42.811799 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:42.812066 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:42.812356 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:42.812361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:42.812363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:42.812499 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:42.812914 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:42.812921 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:42.813159 (+ 238us) client_negotiation.cc:769] Sending connection context
0926 20:24:42.813361 (+ 202us) client_negotiation.cc:240] Negotiation successful
0926 20:24:42.813684 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":193,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.195@KRBTEST.COM" created.
I20260926 20:24:42.815783 18919 heartbeater.cc:347] Connected to a master server at 127.0.116.254:40713
I20260926 20:24:42.816035 18919 heartbeater.cc:464] Registering TS with master...
I20260926 20:24:42.816525 18919 heartbeater.cc:511] Master 127.0.116.254:40713 requested a full tablet report, sending...
I20260926 20:24:42.817721 18594 ts_manager.cc:194] Registered new tserver with Master: e3e2e94c5ec84483ab1a7b48149d055f (127.0.116.194:46879)
I20260926 20:24:42.818321 18594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:53511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
I20260926 20:24:42.831820 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:40713
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:24:42.946826 18924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:42.947147 18924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:42.947225 18924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:42.951743 18924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:42.951828 18924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:42.951921 18924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:24:42.957356 18924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:40713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:42.958546 18924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:42.959772 18924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:42.969560 18929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:42.969616 18930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:42.969574 18924 server_base.cc:1034] running on GCE node
W20260926 20:24:42.969560 18932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:42.970252 18924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:42.970896 18924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:42.972060 18924 hybrid_clock.cc:648] HybridClock initialized: now 1790454282972042 us; error 33 us; skew 500 ppm
Sep 26 20:24:42 dist-test-slave-qsm9 krb5kdc[18549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:42.974995 18924 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:24:42.975920 18924 webserver.cc:460] Webserver started at http://127.0.116.195:33205/ using document root <none> and password file <none>
I20260926 20:24:42.976440 18924 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:42.976496 18924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:42.976666 18924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:42.978739 18924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "38566290af004eec984fe26747abe483"
format_stamp: "Formatted at 2026-09-26 20:24:42 on dist-test-slave-qsm9"
server_key: "e14ca445abec8ea4ede5d3a7d3379a28"
server_key_iv: "3df1ffa82822331b11b1084a30c09393"
server_key_version: "encryptionkey@0"
I20260926 20:24:42.979286 18924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "38566290af004eec984fe26747abe483"
format_stamp: "Formatted at 2026-09-26 20:24:42 on dist-test-slave-qsm9"
server_key: "e14ca445abec8ea4ede5d3a7d3379a28"
server_key_iv: "3df1ffa82822331b11b1084a30c09393"
server_key_version: "encryptionkey@0"
I20260926 20:24:42.982975 18924 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:24:42.985502 18939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:42.986783 18924 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:24:42.986958 18924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "38566290af004eec984fe26747abe483"
format_stamp: "Formatted at 2026-09-26 20:24:42 on dist-test-slave-qsm9"
server_key: "e14ca445abec8ea4ede5d3a7d3379a28"
server_key_iv: "3df1ffa82822331b11b1084a30c09393"
server_key_version: "encryptionkey@0"
I20260926 20:24:42.987078 18924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:43.002626 18924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:43.006163 18924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:43.006381 18924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:43.007005 18924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:43.007886 18924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:24:43.007972 18924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:43.008030 18924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:24:43.008059 18924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:43.017251 18924 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:34023
I20260926 20:24:43.017267 19052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:34023 every 8 connection(s)
I20260926 20:24:43.018190 18924 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:24:43.018370 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 18924
I20260926 20:24:43.018518 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:24:43.018886 467 external_mini_cluster.cc:1428] Setting key cb668e6f81c6a48ec7cff98df91db002
Sep 26 20:24:43 dist-test-slave-qsm9 krb5kdc[18549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:24:43.032358 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:43.020498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:57231 (local address 127.0.116.254:40713)
0926 20:24:43.020653 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:43.020658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:43.021735 (+ 1077us) server_negotiation.cc:407] Connection header received
0926 20:24:43.022993 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:43.022995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:43.023050 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:43.023124 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:43.024293 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.024807 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.025906 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.026126 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.026203 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:43.028702 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:43.028705 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:43.028707 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:43.028732 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:43.030217 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:43.030711 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:43.030714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:43.030716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:43.030828 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:43.031197 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:43.031200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:43.031201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:43.031410 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:43.031563 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:43.032062 (+ 499us) server_negotiation.cc:299] Negotiation successful
0926 20:24:43.032186 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:24:43.033317 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:43.021060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:40713 (local address 127.0.116.195:57231)
0926 20:24:43.021546 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:43.021599 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:43.022786 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:43.023253 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:43.023259 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:43.023694 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:43.024169 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:43.024182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.024947 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.024949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:43.025797 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:43.025805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.026251 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.026254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:43.026386 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:43.027022 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:24:43.027038 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:43.028543 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:43.030349 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:43.030353 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:43.030366 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:43.030616 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:43.030947 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:43.030949 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:43.030950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:43.031088 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:43.031553 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:43.031558 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:43.031786 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 20:24:43.032020 (+ 234us) client_negotiation.cc:240] Negotiation successful
0926 20:24:43.032336 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":134,"threads_started":1}
I20260926 20:24:43.034327 19053 heartbeater.cc:347] Connected to a master server at 127.0.116.254:40713
I20260926 20:24:43.034564 19053 heartbeater.cc:464] Registering TS with master...
I20260926 20:24:43.035054 19053 heartbeater.cc:511] Master 127.0.116.254:40713 requested a full tablet report, sending...
I20260926 20:24:43.036074 18594 ts_manager.cc:194] Registered new tserver with Master: 38566290af004eec984fe26747abe483 (127.0.116.195:34023)
I20260926 20:24:43.036619 18594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:57231
I20260926 20:24:43.045217 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:24:43.055683 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:43.047278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.0.116.254:40713)
0926 20:24:43.047427 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:43.047432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:43.047693 (+ 261us) server_negotiation.cc:407] Connection header received
0926 20:24:43.047849 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:43.047853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:43.047947 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:43.048055 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:43.048845 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.049371 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.050247 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.050475 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.050583 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:43.052269 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:43.052273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:43.052276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:43.052310 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:43.053803 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:43.054329 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:43.054335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:43.054341 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:43.054416 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:43.054741 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:43.054744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:43.054746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:43.054963 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:43.055122 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:43.055394 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 20:24:43.055517 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:24:43.058701 18594 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:24:43.061831 18594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:24:43.077868 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:43.071263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:43149 (local address 127.0.0.1:33298)
0926 20:24:43.072267 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:43.072325 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:43.072517 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:43.073306 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:43.073308 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:43.073328 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:43.073424 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:43.073429 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.074685 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.074688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:43.076331 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:43.076343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.077260 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.077263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:43.077370 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:43.077402 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 20:24:43.077553 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 20:24:43.077691 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":43,"thread_start_us":101,"threads_started":1}
I20260926 20:24:43.077905 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:43.071460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33298 (local address 127.0.116.193:43149)
0926 20:24:43.071901 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:43.071921 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:43.072428 (+ 507us) server_negotiation.cc:407] Connection header received
0926 20:24:43.072718 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:43.072725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:43.072967 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:43.073109 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:43.073540 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.074540 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.076479 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.077127 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.077229 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:43.077360 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:43.077620 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 20:24:43.077732 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":90,"threads_started":1}
I20260926 20:24:43.078727 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:43.071477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:34023 (local address 127.0.0.1:33310)
0926 20:24:43.072369 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:43.072384 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:43.072515 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:43.073111 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:43.073117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:43.073139 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:43.073285 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:43.073294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.074975 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.074979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:43.076337 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:43.076343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.078435 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.078438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:43.078504 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:43.078523 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 20:24:43.078571 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 20:24:43.078624 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":146,"thread_start_us":132,"threads_started":1}
I20260926 20:24:43.078881 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:43.072137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.0.116.195:34023)
0926 20:24:43.072624 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:43.072638 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:43.072660 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:24:43.072733 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:43.072738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:43.072953 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:43.073118 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:43.073416 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.074845 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.077418 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.078292 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.078434 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:43.078592 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:43.078646 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 20:24:43.078736 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":168,"threads_started":1}
I20260926 20:24:43.080364 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:43.072101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:46879 (local address 127.0.0.1:53878)
0926 20:24:43.072968 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:43.072984 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:43.073080 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:43.073631 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:43.073634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:43.073652 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:43.073728 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:43.073733 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.075249 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.075254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:43.076339 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:43.076350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.079944 (+ 3594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.079948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:43.080048 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:43.080073 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:24:43.080141 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 20:24:43.080222 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":59,"thread_start_us":108,"threads_started":1}
I20260926 20:24:43.080418 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:43.072543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53878 (local address 127.0.116.194:46879)
0926 20:24:43.072865 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:43.072876 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:43.073179 (+ 303us) server_negotiation.cc:407] Connection header received
0926 20:24:43.073263 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:43.073270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:43.073490 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:43.073652 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:43.073813 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.075114 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.079063 (+ 3949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.079804 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.079942 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:43.080094 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:43.080165 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 20:24:43.080256 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":134,"threads_started":1}
I20260926 20:24:43.081260 18719 tablet_service.cc:1467] Processing CreateTablet for tablet 2a36002757364e68979da2b3da14d58d (DEFAULT_TABLE table=test-table [id=a1fab1e501294d92933b05477c13023c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:24:43.082458 18719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a36002757364e68979da2b3da14d58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:43.083328 18987 tablet_service.cc:1467] Processing CreateTablet for tablet 2a36002757364e68979da2b3da14d58d (DEFAULT_TABLE table=test-table [id=a1fab1e501294d92933b05477c13023c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:24:43.083407 18853 tablet_service.cc:1467] Processing CreateTablet for tablet 2a36002757364e68979da2b3da14d58d (DEFAULT_TABLE table=test-table [id=a1fab1e501294d92933b05477c13023c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:24:43.084290 18987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a36002757364e68979da2b3da14d58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:43.084460 18853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a36002757364e68979da2b3da14d58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:43.090577 19070 tablet_bootstrap.cc:492] T 2a36002757364e68979da2b3da14d58d P d1063cbcdd614eacbe780d20e3d97264: Bootstrap starting.
I20260926 20:24:43.091998 19071 tablet_bootstrap.cc:492] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483: Bootstrap starting.
I20260926 20:24:43.092136 19072 tablet_bootstrap.cc:492] T 2a36002757364e68979da2b3da14d58d P e3e2e94c5ec84483ab1a7b48149d055f: Bootstrap starting.
I20260926 20:24:43.093612 19070 tablet_bootstrap.cc:654] T 2a36002757364e68979da2b3da14d58d P d1063cbcdd614eacbe780d20e3d97264: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:43.094200 19072 tablet_bootstrap.cc:654] T 2a36002757364e68979da2b3da14d58d P e3e2e94c5ec84483ab1a7b48149d055f: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:43.094512 19071 tablet_bootstrap.cc:654] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:43.095221 19072 log.cc:824] T 2a36002757364e68979da2b3da14d58d P e3e2e94c5ec84483ab1a7b48149d055f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:43.095232 19070 log.cc:824] T 2a36002757364e68979da2b3da14d58d P d1063cbcdd614eacbe780d20e3d97264: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:43.095789 19071 log.cc:824] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:43.097342 19072 tablet_bootstrap.cc:492] T 2a36002757364e68979da2b3da14d58d P e3e2e94c5ec84483ab1a7b48149d055f: No bootstrap required, opened a new log
I20260926 20:24:43.097342 19070 tablet_bootstrap.cc:492] T 2a36002757364e68979da2b3da14d58d P d1063cbcdd614eacbe780d20e3d97264: No bootstrap required, opened a new log
I20260926 20:24:43.097548 19072 ts_tablet_manager.cc:1397] T 2a36002757364e68979da2b3da14d58d P e3e2e94c5ec84483ab1a7b48149d055f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:24:43.097558 19070 ts_tablet_manager.cc:1397] T 2a36002757364e68979da2b3da14d58d P d1063cbcdd614eacbe780d20e3d97264: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 20:24:43.097710 19071 tablet_bootstrap.cc:492] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483: No bootstrap required, opened a new log
I20260926 20:24:43.097937 19071 ts_tablet_manager.cc:1397] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:24:43.101819 19071 raft_consensus.cc:348] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1063cbcdd614eacbe780d20e3d97264" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43149 } } peers { permanent_uuid: "e3e2e94c5ec84483ab1a7b48149d055f" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 46879 } } peers { permanent_uuid: "38566290af004eec984fe26747abe483" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 34023 } }
I20260926 20:24:43.101821 19070 raft_consensus.cc:348] T 2a36002757364e68979da2b3da14d58d P d1063cbcdd614eacbe780d20e3d97264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1063cbcdd614eacbe780d20e3d97264" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43149 } } peers { permanent_uuid: "e3e2e94c5ec84483ab1a7b48149d055f" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 46879 } } peers { permanent_uuid: "38566290af004eec984fe26747abe483" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 34023 } }
I20260926 20:24:43.101830 19072 raft_consensus.cc:348] T 2a36002757364e68979da2b3da14d58d P e3e2e94c5ec84483ab1a7b48149d055f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1063cbcdd614eacbe780d20e3d97264" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43149 } } peers { permanent_uuid: "e3e2e94c5ec84483ab1a7b48149d055f" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 46879 } } peers { permanent_uuid: "38566290af004eec984fe26747abe483" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 34023 } }
I20260926 20:24:43.102169 19072 raft_consensus.cc:374] T 2a36002757364e68979da2b3da14d58d P e3e2e94c5ec84483ab1a7b48149d055f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:43.102169 19070 raft_consensus.cc:374] T 2a36002757364e68979da2b3da14d58d P d1063cbcdd614eacbe780d20e3d97264 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:43.102171 19071 raft_consensus.cc:374] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:43.102231 19071 raft_consensus.cc:729] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38566290af004eec984fe26747abe483, State: Initialized, Role: FOLLOWER
I20260926 20:24:43.102231 19070 raft_consensus.cc:729] T 2a36002757364e68979da2b3da14d58d P d1063cbcdd614eacbe780d20e3d97264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1063cbcdd614eacbe780d20e3d97264, State: Initialized, Role: FOLLOWER
I20260926 20:24:43.102308 19072 raft_consensus.cc:729] T 2a36002757364e68979da2b3da14d58d P e3e2e94c5ec84483ab1a7b48149d055f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3e2e94c5ec84483ab1a7b48149d055f, State: Initialized, Role: FOLLOWER
I20260926 20:24:43.102757 19071 consensus_queue.cc:260] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1063cbcdd614eacbe780d20e3d97264" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43149 } } peers { permanent_uuid: "e3e2e94c5ec84483ab1a7b48149d055f" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 46879 } } peers { permanent_uuid: "38566290af004eec984fe26747abe483" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 34023 } }
I20260926 20:24:43.102762 19070 consensus_queue.cc:260] T 2a36002757364e68979da2b3da14d58d P d1063cbcdd614eacbe780d20e3d97264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1063cbcdd614eacbe780d20e3d97264" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43149 } } peers { permanent_uuid: "e3e2e94c5ec84483ab1a7b48149d055f" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 46879 } } peers { permanent_uuid: "38566290af004eec984fe26747abe483" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 34023 } }
I20260926 20:24:43.102762 19072 consensus_queue.cc:260] T 2a36002757364e68979da2b3da14d58d P e3e2e94c5ec84483ab1a7b48149d055f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1063cbcdd614eacbe780d20e3d97264" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43149 } } peers { permanent_uuid: "e3e2e94c5ec84483ab1a7b48149d055f" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 46879 } } peers { permanent_uuid: "38566290af004eec984fe26747abe483" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 34023 } }
I20260926 20:24:43.103528 19053 heartbeater.cc:503] Master 127.0.116.254:40713 was elected leader, sending a full tablet report...
I20260926 20:24:43.103906 19072 ts_tablet_manager.cc:1428] T 2a36002757364e68979da2b3da14d58d P e3e2e94c5ec84483ab1a7b48149d055f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:24:43.103906 19070 ts_tablet_manager.cc:1428] T 2a36002757364e68979da2b3da14d58d P d1063cbcdd614eacbe780d20e3d97264: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:24:43.104146 18785 heartbeater.cc:503] Master 127.0.116.254:40713 was elected leader, sending a full tablet report...
I20260926 20:24:43.105167 19071 ts_tablet_manager.cc:1428] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 20:24:43.105171 18919 heartbeater.cc:503] Master 127.0.116.254:40713 was elected leader, sending a full tablet report...
W20260926 20:24:43.269549 19054 tablet.cc:2369] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:24:43.300460 18920 tablet.cc:2369] T 2a36002757364e68979da2b3da14d58d P e3e2e94c5ec84483ab1a7b48149d055f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:24:43.331120 18786 tablet.cc:2369] T 2a36002757364e68979da2b3da14d58d P d1063cbcdd614eacbe780d20e3d97264: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:24:43.349550 19078 raft_consensus.cc:482] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:24:43.349802 19078 raft_consensus.cc:504] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1063cbcdd614eacbe780d20e3d97264" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43149 } } peers { permanent_uuid: "e3e2e94c5ec84483ab1a7b48149d055f" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 46879 } } peers { permanent_uuid: "38566290af004eec984fe26747abe483" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 34023 } }
I20260926 20:24:43.351027 19078 leader_election.cc:288] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1063cbcdd614eacbe780d20e3d97264 (127.0.116.193:43149), e3e2e94c5ec84483ab1a7b48149d055f (127.0.116.194:46879)
I20260926 20:24:43.354691 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:43.351436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:48655 (local address 127.0.116.193:43149)
0926 20:24:43.351599 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:43.351604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:43.351631 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:24:43.351714 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:43.351719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:43.351781 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:43.351862 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:43.352144 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.352654 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.353695 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.354129 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.354260 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:43.354315 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:43.354496 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 20:24:43.354560 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 20:24:43.354689 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:43.351273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:43149 (local address 127.0.116.195:48655)
0926 20:24:43.351430 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:43.351444 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:43.351606 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:43.351895 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:43.351898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:43.351916 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:43.352033 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:43.352039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.352766 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.352769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:43.353571 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:43.353580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.354250 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.354254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:43.354360 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:43.354386 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 20:24:43.354480 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 20:24:43.354554 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 20:24:43.355113 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:43.351740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:55813 (local address 127.0.116.194:46879)
0926 20:24:43.351892 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:43.351897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:43.352080 (+ 183us) server_negotiation.cc:407] Connection header received
0926 20:24:43.352230 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:43.352233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:43.352283 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:43.352366 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:43.352622 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.353104 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.354162 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.354563 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.354690 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:43.354736 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:43.354946 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 20:24:43.355025 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 20:24:43.355476 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:43.351603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:46879 (local address 127.0.116.195:55813)
0926 20:24:43.351992 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:43.352003 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:43.352118 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:43.352376 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:43.352379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:43.352394 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:43.352516 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:43.352521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.353215 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.353218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:43.354039 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:43.354047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.354681 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.354684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:43.354790 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:43.354819 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 20:24:43.355274 (+ 455us) client_negotiation.cc:240] Negotiation successful
0926 20:24:43.355364 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":120,"threads_started":1}
I20260926 20:24:43.355428 18739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a36002757364e68979da2b3da14d58d" candidate_uuid: "38566290af004eec984fe26747abe483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1063cbcdd614eacbe780d20e3d97264" is_pre_election: true
I20260926 20:24:43.355816 18739 raft_consensus.cc:2454] T 2a36002757364e68979da2b3da14d58d P d1063cbcdd614eacbe780d20e3d97264 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38566290af004eec984fe26747abe483 in term 0.
I20260926 20:24:43.356170 18873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a36002757364e68979da2b3da14d58d" candidate_uuid: "38566290af004eec984fe26747abe483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3e2e94c5ec84483ab1a7b48149d055f" is_pre_election: true
I20260926 20:24:43.356370 18943 leader_election.cc:302] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38566290af004eec984fe26747abe483, d1063cbcdd614eacbe780d20e3d97264; no voters:
I20260926 20:24:43.356487 18873 raft_consensus.cc:2454] T 2a36002757364e68979da2b3da14d58d P e3e2e94c5ec84483ab1a7b48149d055f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38566290af004eec984fe26747abe483 in term 0.
I20260926 20:24:43.356683 19078 raft_consensus.cc:2781] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:24:43.356777 19078 raft_consensus.cc:482] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:24:43.356838 19078 raft_consensus.cc:3037] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:43.357849 19078 raft_consensus.cc:504] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1063cbcdd614eacbe780d20e3d97264" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43149 } } peers { permanent_uuid: "e3e2e94c5ec84483ab1a7b48149d055f" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 46879 } } peers { permanent_uuid: "38566290af004eec984fe26747abe483" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 34023 } }
I20260926 20:24:43.358135 19078 leader_election.cc:288] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483 [CANDIDATE]: Term 1 election: Requested vote from peers d1063cbcdd614eacbe780d20e3d97264 (127.0.116.193:43149), e3e2e94c5ec84483ab1a7b48149d055f (127.0.116.194:46879)
I20260926 20:24:43.358639 18739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a36002757364e68979da2b3da14d58d" candidate_uuid: "38566290af004eec984fe26747abe483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1063cbcdd614eacbe780d20e3d97264"
I20260926 20:24:43.358709 18873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a36002757364e68979da2b3da14d58d" candidate_uuid: "38566290af004eec984fe26747abe483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3e2e94c5ec84483ab1a7b48149d055f"
I20260926 20:24:43.358780 18739 raft_consensus.cc:3037] T 2a36002757364e68979da2b3da14d58d P d1063cbcdd614eacbe780d20e3d97264 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:43.358820 18873 raft_consensus.cc:3037] T 2a36002757364e68979da2b3da14d58d P e3e2e94c5ec84483ab1a7b48149d055f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:43.360159 18873 raft_consensus.cc:2454] T 2a36002757364e68979da2b3da14d58d P e3e2e94c5ec84483ab1a7b48149d055f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38566290af004eec984fe26747abe483 in term 1.
I20260926 20:24:43.360159 18739 raft_consensus.cc:2454] T 2a36002757364e68979da2b3da14d58d P d1063cbcdd614eacbe780d20e3d97264 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38566290af004eec984fe26747abe483 in term 1.
I20260926 20:24:43.360458 18943 leader_election.cc:302] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38566290af004eec984fe26747abe483, d1063cbcdd614eacbe780d20e3d97264; no voters:
I20260926 20:24:43.360628 19078 raft_consensus.cc:2781] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:24:43.360899 19078 raft_consensus.cc:686] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483 [term 1 LEADER]: Becoming Leader. State: Replica: 38566290af004eec984fe26747abe483, State: Running, Role: LEADER
I20260926 20:24:43.361408 19078 consensus_queue.cc:237] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1063cbcdd614eacbe780d20e3d97264" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43149 } } peers { permanent_uuid: "e3e2e94c5ec84483ab1a7b48149d055f" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 46879 } } peers { permanent_uuid: "38566290af004eec984fe26747abe483" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 34023 } }
I20260926 20:24:43.364768 18592 catalog_manager.cc:5495] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38566290af004eec984fe26747abe483 (127.0.116.195). New cstate: current_term: 1 leader_uuid: "38566290af004eec984fe26747abe483" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1063cbcdd614eacbe780d20e3d97264" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3e2e94c5ec84483ab1a7b48149d055f" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 46879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38566290af004eec984fe26747abe483" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 34023 } health_report { overall_health: HEALTHY } } }
I20260926 20:24:43.444375 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:43.440167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.0.116.195:34023)
0926 20:24:43.440334 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:43.440340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:43.440357 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:24:43.440499 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:43.440501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:43.440557 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:43.440684 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:24:43.440964 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.441525 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.442577 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.442871 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.442996 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:43.443372 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:24:43.443861 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:24:43.444041 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:43.444136 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 20:24:43.444181 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 20:24:43.451330 18873 raft_consensus.cc:1261] T 2a36002757364e68979da2b3da14d58d P e3e2e94c5ec84483ab1a7b48149d055f [term 1 FOLLOWER]: Refusing update from remote peer 38566290af004eec984fe26747abe483: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:24:43.451330 18739 raft_consensus.cc:1261] T 2a36002757364e68979da2b3da14d58d P d1063cbcdd614eacbe780d20e3d97264 [term 1 FOLLOWER]: Refusing update from remote peer 38566290af004eec984fe26747abe483: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:24:43.452241 19080 consensus_queue.cc:1035] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3e2e94c5ec84483ab1a7b48149d055f" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 46879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:24:43.452477 19078 consensus_queue.cc:1035] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1063cbcdd614eacbe780d20e3d97264" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 43149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:24:43.459901 19083 mvcc.cc:205] Tried to move back new op lower bound from 7333700745007898624 to 7333700744651456512. Current Snapshot: MvccSnapshot[applied={T|T < 7333700745007898624}]
I20260926 20:24:43.460235 19085 mvcc.cc:205] Tried to move back new op lower bound from 7333700745007898624 to 7333700744651456512. Current Snapshot: MvccSnapshot[applied={T|T < 7333700745007898624}]
I20260926 20:24:43.470693 18593 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:24:43.470870 18593 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:24:43.473601 18593 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 721942f8ae9e4dd1a29ce024011b6340: Sending DeleteTablet for 3 replicas of tablet 2a36002757364e68979da2b3da14d58d
I20260926 20:24:43.474604 18987 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a36002757364e68979da2b3da14d58d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:24:43 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:33310
I20260926 20:24:43.474624 18719 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a36002757364e68979da2b3da14d58d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:24:43 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:33298
I20260926 20:24:43.475169 19094 tablet_replica.cc:330] T 2a36002757364e68979da2b3da14d58d P d1063cbcdd614eacbe780d20e3d97264: stopping tablet replica
I20260926 20:24:43.475169 19093 tablet_replica.cc:330] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483: stopping tablet replica
I20260926 20:24:43.475113 18853 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a36002757364e68979da2b3da14d58d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:24:43 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:53878
I20260926 20:24:43.475543 19093 raft_consensus.cc:2229] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:24:43.475572 19094 raft_consensus.cc:2229] T 2a36002757364e68979da2b3da14d58d P d1063cbcdd614eacbe780d20e3d97264 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:24:43.475904 19094 raft_consensus.cc:2258] T 2a36002757364e68979da2b3da14d58d P d1063cbcdd614eacbe780d20e3d97264 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:24:43.475927 19093 raft_consensus.cc:2258] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:24:43.476425 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 18656
I20260926 20:24:43.476284 19095 tablet_replica.cc:330] T 2a36002757364e68979da2b3da14d58d P e3e2e94c5ec84483ab1a7b48149d055f: stopping tablet replica
I20260926 20:24:43.477756 19095 raft_consensus.cc:2229] T 2a36002757364e68979da2b3da14d58d P e3e2e94c5ec84483ab1a7b48149d055f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:24:43.478078 19095 raft_consensus.cc:2258] T 2a36002757364e68979da2b3da14d58d P e3e2e94c5ec84483ab1a7b48149d055f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:24:43.478387 19093 ts_tablet_manager.cc:1905] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:24:43.479748 19095 ts_tablet_manager.cc:1905] T 2a36002757364e68979da2b3da14d58d P e3e2e94c5ec84483ab1a7b48149d055f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:24:43.481349 19093 ts_tablet_manager.cc:1918] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:24:43.481460 19093 log.cc:1196] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/2a36002757364e68979da2b3da14d58d
I20260926 20:24:43.481802 19093 ts_tablet_manager.cc:1939] T 2a36002757364e68979da2b3da14d58d P 38566290af004eec984fe26747abe483: Deleting consensus metadata
I20260926 20:24:43.482604 19095 ts_tablet_manager.cc:1918] T 2a36002757364e68979da2b3da14d58d P e3e2e94c5ec84483ab1a7b48149d055f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:24:43.482704 19095 log.cc:1196] T 2a36002757364e68979da2b3da14d58d P e3e2e94c5ec84483ab1a7b48149d055f: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/2a36002757364e68979da2b3da14d58d
I20260926 20:24:43.483024 19095 ts_tablet_manager.cc:1939] T 2a36002757364e68979da2b3da14d58d P e3e2e94c5ec84483ab1a7b48149d055f: Deleting consensus metadata
I20260926 20:24:43.483198 18578 catalog_manager.cc:4841] TS 38566290af004eec984fe26747abe483 (127.0.116.195:34023): tablet 2a36002757364e68979da2b3da14d58d (table test-table [id=a1fab1e501294d92933b05477c13023c]) successfully deleted
I20260926 20:24:43.484406 18581 catalog_manager.cc:4841] TS e3e2e94c5ec84483ab1a7b48149d055f (127.0.116.194:46879): tablet 2a36002757364e68979da2b3da14d58d (table test-table [id=a1fab1e501294d92933b05477c13023c]) successfully deleted
W20260926 20:24:43.484913 18581 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.116.193:43149 (error 108)
I20260926 20:24:43.485028 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 18790
I20260926 20:24:43.486447 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:43.486177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:43149 (local address 127.0.0.1:33306)
0926 20:24:43.486310 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:43.486375 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.193:43149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 20:24:43.486635 18581 catalog_manager.cc:4584] TS d1063cbcdd614eacbe780d20e3d97264 (127.0.116.193:43149): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a36002757364e68979da2b3da14d58d: Network error: Client connection negotiation failed: client connection to 127.0.116.193:43149: connect: Connection refused (error 111)
I20260926 20:24:43.492617 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 18924
I20260926 20:24:43.498494 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 18562
2026-09-26T20:24:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2622 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T20:24:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:24:43Z Disabled control of system clock
I20260926 20:24:43.529548 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:34181
--webserver_interface=127.0.116.254
--webserver_port=0
--builtin_ntp_servers=127.0.116.212:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:34181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:24:43.640671 19100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:43.640951 19100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:43.640988 19100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:43.645172 19100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:24:43.645234 19100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:43.645249 19100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:43.645267 19100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:24:43.645282 19100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:24:43.650157 19100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:34181
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:34181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:43.651335 19100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:43.652804 19100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:43.661641 19105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:43.661641 19108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:43.662012 19100 server_base.cc:1034] running on GCE node
W20260926 20:24:43.662194 19106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:43.662786 19100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:43.663996 19100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:43.665150 19100 hybrid_clock.cc:648] HybridClock initialized: now 1790454283665135 us; error 34 us; skew 500 ppm
I20260926 20:24:43.667057 19100 webserver.cc:460] Webserver started at http://127.0.116.254:39281/ using document root <none> and password file <none>
I20260926 20:24:43.667735 19100 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:43.667802 19100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:43.668030 19100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:43.670121 19100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "066979cdee9f47b0a91c67944d3cca56"
format_stamp: "Formatted at 2026-09-26 20:24:43 on dist-test-slave-qsm9"
server_key: "4b91f8fa350d0e8258032bccf4852be9"
server_key_iv: "04eb001ab3c3004c8005bb94224e6e6a"
server_key_version: "encryptionkey@0"
I20260926 20:24:43.670696 19100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "066979cdee9f47b0a91c67944d3cca56"
format_stamp: "Formatted at 2026-09-26 20:24:43 on dist-test-slave-qsm9"
server_key: "4b91f8fa350d0e8258032bccf4852be9"
server_key_iv: "04eb001ab3c3004c8005bb94224e6e6a"
server_key_version: "encryptionkey@0"
I20260926 20:24:43.674616 19100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:24:43.677243 19114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:43.678571 19100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:24:43.678673 19100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "066979cdee9f47b0a91c67944d3cca56"
format_stamp: "Formatted at 2026-09-26 20:24:43 on dist-test-slave-qsm9"
server_key: "4b91f8fa350d0e8258032bccf4852be9"
server_key_iv: "04eb001ab3c3004c8005bb94224e6e6a"
server_key_version: "encryptionkey@0"
I20260926 20:24:43.678813 19100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:43.695114 19100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:43.695956 19100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:43.696158 19100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:43.703140 19100 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:34181
I20260926 20:24:43.703150 19166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:34181 every 8 connection(s)
I20260926 20:24:43.704164 19100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:24:43.705678 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 19100
I20260926 20:24:43.705823 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:24:43.706121 467 external_mini_cluster.cc:1428] Setting key 61bbd2d01f2724a8722901e6deaf01c3
I20260926 20:24:43.707934 19167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:43.714236 19167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56: Bootstrap starting.
I20260926 20:24:43.715317 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:43.707648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37188 (local address 127.0.116.254:34181)
0926 20:24:43.708261 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:43.708277 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:43.708312 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:24:43.708978 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:43.708995 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:43.709372 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:43.709672 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:43.710011 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.710968 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.712158 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.712612 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.712725 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:43.713633 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:43.713638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:43.713650 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:43.714031 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:43.714085 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:43.714103 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:43.714178 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:43.714372 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 20:24:43.714629 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":134,"threads_started":1}
I20260926 20:24:43.716756 19167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:43.717664 19167 log.cc:824] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:43.719709 19167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56: No bootstrap required, opened a new log
I20260926 20:24:43.723338 19167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "066979cdee9f47b0a91c67944d3cca56" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34181 } }
I20260926 20:24:43.723680 19167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:43.723726 19167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 066979cdee9f47b0a91c67944d3cca56, State: Initialized, Role: FOLLOWER
I20260926 20:24:43.724231 19167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "066979cdee9f47b0a91c67944d3cca56" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34181 } }
I20260926 20:24:43.724412 19167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:24:43.724488 19167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:24:43.724586 19167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:43.725489 19167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "066979cdee9f47b0a91c67944d3cca56" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34181 } }
I20260926 20:24:43.725785 19167 leader_election.cc:302] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 066979cdee9f47b0a91c67944d3cca56; no voters:
I20260926 20:24:43.726092 19167 leader_election.cc:288] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:24:43.726195 19172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:24:43.726385 19172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56 [term 1 LEADER]: Becoming Leader. State: Replica: 066979cdee9f47b0a91c67944d3cca56, State: Running, Role: LEADER
I20260926 20:24:43.726696 19172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "066979cdee9f47b0a91c67944d3cca56" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34181 } }
I20260926 20:24:43.727452 19167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:24:43.728524 19174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 066979cdee9f47b0a91c67944d3cca56. Latest consensus state: current_term: 1 leader_uuid: "066979cdee9f47b0a91c67944d3cca56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "066979cdee9f47b0a91c67944d3cca56" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34181 } } }
I20260926 20:24:43.728662 19174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56 [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:43.728950 19173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "066979cdee9f47b0a91c67944d3cca56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "066979cdee9f47b0a91c67944d3cca56" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34181 } } }
I20260926 20:24:43.729044 19173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56 [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:43.729144 19181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:24:43.732488 19181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:24:43.738252 19181 catalog_manager.cc:1347] Generated new cluster ID: 52afa1ad8aed41ec9f82e25f6e51cf64
I20260926 20:24:43.738339 19181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:24:43.747902 19181 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:24:43.749183 19181 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:24:43.761014 19181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56: Generated new TSK 0
I20260926 20:24:43.761919 19181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:24:43.766356 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:34181
--builtin_ntp_servers=127.0.116.212:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:24:43.881441 19191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:43.881798 19191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:43.881847 19191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:43.886168 19191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:43.886240 19191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:43.886296 19191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:24:43.891255 19191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:34181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:43.892517 19191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:43.893882 19191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:43.904072 19199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:43.904062 19197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:43.904062 19196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:43.904402 19191 server_base.cc:1034] running on GCE node
I20260926 20:24:43.904809 19191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:43.905494 19191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:43.906668 19191 hybrid_clock.cc:648] HybridClock initialized: now 1790454283906643 us; error 43 us; skew 500 ppm
I20260926 20:24:43.908879 19191 webserver.cc:460] Webserver started at http://127.0.116.193:41163/ using document root <none> and password file <none>
I20260926 20:24:43.909514 19191 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:43.909581 19191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:43.909813 19191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:43.912065 19191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "ffa2f0eb183945419b3689c5e5612166"
format_stamp: "Formatted at 2026-09-26 20:24:43 on dist-test-slave-qsm9"
server_key: "fb48fec4729a45c3bf7e72383fa56ae7"
server_key_iv: "a0a94704c213973f48b866c8eda11663"
server_key_version: "encryptionkey@0"
I20260926 20:24:43.912664 19191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "ffa2f0eb183945419b3689c5e5612166"
format_stamp: "Formatted at 2026-09-26 20:24:43 on dist-test-slave-qsm9"
server_key: "fb48fec4729a45c3bf7e72383fa56ae7"
server_key_iv: "a0a94704c213973f48b866c8eda11663"
server_key_version: "encryptionkey@0"
I20260926 20:24:43.916497 19191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:24:43.919065 19205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:43.920531 19191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:24:43.920699 19191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "ffa2f0eb183945419b3689c5e5612166"
format_stamp: "Formatted at 2026-09-26 20:24:43 on dist-test-slave-qsm9"
server_key: "fb48fec4729a45c3bf7e72383fa56ae7"
server_key_iv: "a0a94704c213973f48b866c8eda11663"
server_key_version: "encryptionkey@0"
I20260926 20:24:43.920830 19191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:43.937278 19191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:43.938030 19191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:43.938239 19191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:43.938876 19191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:43.939879 19191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:24:43.939954 19191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:43.940029 19191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:24:43.940074 19191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:43.949545 19191 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:40669
I20260926 20:24:43.949566 19318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:40669 every 8 connection(s)
I20260926 20:24:43.950479 19191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:24:43.953219 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 19191
I20260926 20:24:43.953354 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:24:43.953652 467 external_mini_cluster.cc:1428] Setting key d162d4ee58b06fe995545812158f40cd
I20260926 20:24:43.955798 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:34181
--builtin_ntp_servers=127.0.116.212:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:24:43.960698 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:43.952345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:56397 (local address 127.0.116.254:34181)
0926 20:24:43.952500 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:43.952504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:43.953780 (+ 1276us) server_negotiation.cc:407] Connection header received
0926 20:24:43.955409 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:43.955413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:43.955487 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:43.955604 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:43.956606 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.957411 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.958592 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.958828 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.958944 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:43.960174 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:43.960178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:43.960180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:43.960195 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:43.960224 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:43.960232 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:43.960308 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:43.960476 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 20:24:43.960550 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:24:43.961580 19321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:43.952741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:34181 (local address 127.0.116.193:56397)
0926 20:24:43.953585 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:43.953634 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:43.955193 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:43.955786 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:43.955793 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:43.955919 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:43.956458 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:43.956472 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.957567 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.957570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:43.958464 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:43.958474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:43.958991 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:43.958995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:43.959165 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:43.959893 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:24:43.959894 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:43.959938 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:24:43.960013 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:43.960322 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:43.960334 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:43.960343 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 20:24:43.960475 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 20:24:43.960779 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":149,"threads_started":1}
I20260926 20:24:43.962812 19319 heartbeater.cc:347] Connected to a master server at 127.0.116.254:34181
I20260926 20:24:43.963100 19319 heartbeater.cc:464] Registering TS with master...
I20260926 20:24:43.963807 19319 heartbeater.cc:511] Master 127.0.116.254:34181 requested a full tablet report, sending...
I20260926 20:24:43.965477 19131 ts_manager.cc:194] Registered new tserver with Master: ffa2f0eb183945419b3689c5e5612166 (127.0.116.193:40669)
I20260926 20:24:43.966636 19131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.116.193:56397
W20260926 20:24:44.070389 19322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:44.070654 19322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:44.070693 19322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:44.074949 19322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:44.075016 19322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:44.075102 19322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:24:44.080000 19322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:34181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:44.081135 19322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:44.082568 19322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:44.092319 19328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:44.092335 19330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:44.092372 19322 server_base.cc:1034] running on GCE node
W20260926 20:24:44.092334 19327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:44.092973 19322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:44.093644 19322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:44.094830 19322 hybrid_clock.cc:648] HybridClock initialized: now 1790454284094814 us; error 33 us; skew 500 ppm
I20260926 20:24:44.096813 19322 webserver.cc:460] Webserver started at http://127.0.116.194:40095/ using document root <none> and password file <none>
I20260926 20:24:44.097393 19322 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:44.097486 19322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:44.097716 19322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:44.099804 19322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "78277fdb6ad44c7ea2d23976408a837c"
format_stamp: "Formatted at 2026-09-26 20:24:44 on dist-test-slave-qsm9"
server_key: "9c95e4f28cbb74ea0e5e3c17116b97ce"
server_key_iv: "d19d355b5f3a647db1a03940276c8bac"
server_key_version: "encryptionkey@0"
I20260926 20:24:44.100360 19322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "78277fdb6ad44c7ea2d23976408a837c"
format_stamp: "Formatted at 2026-09-26 20:24:44 on dist-test-slave-qsm9"
server_key: "9c95e4f28cbb74ea0e5e3c17116b97ce"
server_key_iv: "d19d355b5f3a647db1a03940276c8bac"
server_key_version: "encryptionkey@0"
I20260926 20:24:44.103982 19322 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:24:44.106411 19336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:44.107754 19322 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:24:44.107887 19322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "78277fdb6ad44c7ea2d23976408a837c"
format_stamp: "Formatted at 2026-09-26 20:24:44 on dist-test-slave-qsm9"
server_key: "9c95e4f28cbb74ea0e5e3c17116b97ce"
server_key_iv: "d19d355b5f3a647db1a03940276c8bac"
server_key_version: "encryptionkey@0"
I20260926 20:24:44.108003 19322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:44.135241 19322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:44.136080 19322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:44.136299 19322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:44.136946 19322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:44.137979 19322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:24:44.138051 19322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:44.138125 19322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:24:44.138170 19322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:44.147239 19322 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:38189
I20260926 20:24:44.147264 19449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:38189 every 8 connection(s)
I20260926 20:24:44.148231 19322 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:24:44.152885 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 19322
I20260926 20:24:44.153023 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:24:44.153348 467 external_mini_cluster.cc:1428] Setting key b6bfced8a6915ec02474163d3b41bde4
I20260926 20:24:44.155740 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:34181
--builtin_ntp_servers=127.0.116.212:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:24:44.159826 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:44.150249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:52807 (local address 127.0.116.254:34181)
0926 20:24:44.150424 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:44.150428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:44.151615 (+ 1187us) server_negotiation.cc:407] Connection header received
0926 20:24:44.153380 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:44.153383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:44.153487 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:44.153580 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:44.154714 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.155313 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.157379 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.157785 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.157974 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:44.159160 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:44.159167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:44.159173 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:44.159193 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:44.159219 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:44.159232 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:44.159312 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:44.159540 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 20:24:44.159617 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 20:24:44.160701 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:44.150700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:34181 (local address 127.0.116.194:52807)
0926 20:24:44.151426 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:44.151471 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:44.153110 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:44.153810 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:44.153817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:44.153922 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:44.154520 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:44.154534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.155529 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.155537 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:44.157193 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:44.157211 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.158011 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.158014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:44.158166 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:44.158863 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:24:44.158865 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:44.158911 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:24:44.158982 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:44.159321 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:44.159328 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:44.159333 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:24:44.159455 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 20:24:44.159795 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":125,"threads_started":1}
I20260926 20:24:44.162063 19450 heartbeater.cc:347] Connected to a master server at 127.0.116.254:34181
I20260926 20:24:44.162348 19450 heartbeater.cc:464] Registering TS with master...
I20260926 20:24:44.163064 19450 heartbeater.cc:511] Master 127.0.116.254:34181 requested a full tablet report, sending...
I20260926 20:24:44.164316 19131 ts_manager.cc:194] Registered new tserver with Master: 78277fdb6ad44c7ea2d23976408a837c (127.0.116.194:38189)
I20260926 20:24:44.164919 19131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.116.194:52807
W20260926 20:24:44.271793 19453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:44.272089 19453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:44.272131 19453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:44.276407 19453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:44.276475 19453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:44.276533 19453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:24:44.281595 19453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:34181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:44.282893 19453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:44.284379 19453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:44.294340 19459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:44.294299 19458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:44.294389 19461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:44.294744 19453 server_base.cc:1034] running on GCE node
I20260926 20:24:44.295295 19453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:44.296077 19453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:44.297297 19453 hybrid_clock.cc:648] HybridClock initialized: now 1790454284297267 us; error 35 us; skew 500 ppm
I20260926 20:24:44.299677 19453 webserver.cc:460] Webserver started at http://127.0.116.195:44153/ using document root <none> and password file <none>
I20260926 20:24:44.300405 19453 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:44.300491 19453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:44.300704 19453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:44.303194 19453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "359d3fcb60cc4321be73ebccb93775ec"
format_stamp: "Formatted at 2026-09-26 20:24:44 on dist-test-slave-qsm9"
server_key: "08e32149f4e7548bf6a868f32c3af17c"
server_key_iv: "453629e3d2826c9b4bcbc62afed752e8"
server_key_version: "encryptionkey@0"
I20260926 20:24:44.303812 19453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "359d3fcb60cc4321be73ebccb93775ec"
format_stamp: "Formatted at 2026-09-26 20:24:44 on dist-test-slave-qsm9"
server_key: "08e32149f4e7548bf6a868f32c3af17c"
server_key_iv: "453629e3d2826c9b4bcbc62afed752e8"
server_key_version: "encryptionkey@0"
I20260926 20:24:44.307583 19453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 20:24:44.310031 19467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:44.311335 19453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:24:44.311467 19453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "359d3fcb60cc4321be73ebccb93775ec"
format_stamp: "Formatted at 2026-09-26 20:24:44 on dist-test-slave-qsm9"
server_key: "08e32149f4e7548bf6a868f32c3af17c"
server_key_iv: "453629e3d2826c9b4bcbc62afed752e8"
server_key_version: "encryptionkey@0"
I20260926 20:24:44.311648 19453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:44.327826 19453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:44.328660 19453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:44.328866 19453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:44.329525 19453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:44.330453 19453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:24:44.330520 19453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:44.330587 19453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:24:44.330637 19453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:44.339783 19453 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:35663
I20260926 20:24:44.339807 19580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:35663 every 8 connection(s)
I20260926 20:24:44.340698 19453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:24:44.342924 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 19453
I20260926 20:24:44.343036 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:24:44.343327 467 external_mini_cluster.cc:1428] Setting key 22c90b63decd7ea1dc8242d90610db56
I20260926 20:24:44.350394 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:44.342511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:45075 (local address 127.0.116.254:34181)
0926 20:24:44.342658 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:44.342661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:44.344032 (+ 1371us) server_negotiation.cc:407] Connection header received
0926 20:24:44.345507 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:44.345510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:44.345568 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:44.345658 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:44.346633 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.347158 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.348399 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.348618 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.348714 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:44.349958 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:44.349961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:44.349963 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:44.349976 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:44.349994 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:44.350002 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:44.350063 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:44.350214 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 20:24:44.350276 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:24:44.351281 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:44.342926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:34181 (local address 127.0.116.195:45075)
0926 20:24:44.343692 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:44.343744 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:44.345279 (+ 1535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:44.345918 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:44.345925 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:44.346014 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:44.346505 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:44.346517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.347285 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.347288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:44.348272 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:44.348282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.348770 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.348773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:44.348942 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:44.349658 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:24:44.349660 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:44.349703 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:24:44.349767 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:44.350087 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:44.350093 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:44.350098 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:24:44.350192 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 20:24:44.350417 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":173,"threads_started":1}
I20260926 20:24:44.352471 19581 heartbeater.cc:347] Connected to a master server at 127.0.116.254:34181
I20260926 20:24:44.352721 19581 heartbeater.cc:464] Registering TS with master...
I20260926 20:24:44.353253 19581 heartbeater.cc:511] Master 127.0.116.254:34181 requested a full tablet report, sending...
I20260926 20:24:44.354324 19131 ts_manager.cc:194] Registered new tserver with Master: 359d3fcb60cc4321be73ebccb93775ec (127.0.116.195:35663)
I20260926 20:24:44.354961 19131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.116.195:45075
I20260926 20:24:44.357777 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:24:44.363731 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:44.359963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37204 (local address 127.0.116.254:34181)
0926 20:24:44.360095 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:44.360098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:44.360114 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:24:44.360240 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:44.360242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:44.360279 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:44.360364 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:44.360637 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.361175 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.362003 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.362208 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.362290 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:44.363207 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:44.363210 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:44.363212 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:44.363223 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:44.363244 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:44.363280 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:44.363378 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:44.363575 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 20:24:44.363625 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:24:44.366659 19131 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:24:44.369709 19131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:24:44.384536 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:44.379262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.0.116.194:38189)
0926 20:24:44.379793 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:44.379803 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:44.379824 (+ 21us) server_negotiation.cc:407] Connection header received
0926 20:24:44.379896 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:44.379901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:44.380075 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:44.380329 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:44.380566 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.381398 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.383150 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.383850 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.383984 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:44.384195 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:44.384265 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 20:24:44.384359 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":141,"threads_started":1}
I20260926 20:24:44.384658 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:44.378577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:40669 (local address 127.0.0.1:49142)
0926 20:24:44.379089 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:44.379102 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:44.379192 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:44.380091 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:44.380097 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:44.380122 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:44.380277 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:44.380309 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.381953 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.381960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:44.383184 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:44.383194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.384225 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.384228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:44.384313 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:44.384346 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 20:24:44.384401 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 20:24:44.384471 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":3,"thread_start_us":114,"threads_started":1}
I20260926 20:24:44.384531 19595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:44.379106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:38189 (local address 127.0.0.1:58206)
0926 20:24:44.379378 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:44.379390 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:44.379470 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:44.380331 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:44.380336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:44.380359 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:44.380452 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:44.380457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.381516 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.381521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:44.382960 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:44.382971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.383983 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.383987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:44.384092 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:44.384129 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 20:24:44.384229 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 20:24:44.384360 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":50,"thread_start_us":87,"threads_started":1}
I20260926 20:24:44.384802 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:44.378370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:35663 (local address 127.0.0.1:47776)
0926 20:24:44.378710 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:44.378766 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:44.378897 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:44.380398 (+ 1501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:44.380401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:44.380419 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:44.380542 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:44.380549 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.381917 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.381920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:44.383136 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:44.383152 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.384442 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.384446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:44.384519 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:44.384541 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:24:44.384605 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 20:24:44.384686 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":67,"thread_start_us":94,"threads_started":1}
I20260926 20:24:44.385252 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:44.378504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47776 (local address 127.0.116.195:35663)
0926 20:24:44.379765 (+ 1261us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:44.379781 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:44.379807 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:24:44.379896 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:44.379901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:44.380075 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:44.380235 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:44.380650 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.381815 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.383255 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.384082 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.384201 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:44.384385 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:44.384977 (+ 592us) server_negotiation.cc:299] Negotiation successful
0926 20:24:44.385102 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1152,"thread_start_us":164,"threads_started":1}
I20260926 20:24:44.385730 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:44.379083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49142 (local address 127.0.116.193:40669)
0926 20:24:44.379552 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:44.379565 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:44.379590 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:24:44.379672 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:44.379678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:44.379906 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:44.380462 (+ 556us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:44.380607 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.381793 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.383301 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.384082 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.385230 (+ 1148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:44.385440 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:44.385491 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 20:24:44.385580 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":124,"threads_started":1}
I20260926 20:24:44.387929 19384 tablet_service.cc:1467] Processing CreateTablet for tablet 06fa46925ef2479d8ef219bf37a9e566 (DEFAULT_TABLE table=test-table [id=c69ac2eeb461427aa5f9ca1d1107cbd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:24:44.388023 19515 tablet_service.cc:1467] Processing CreateTablet for tablet 06fa46925ef2479d8ef219bf37a9e566 (DEFAULT_TABLE table=test-table [id=c69ac2eeb461427aa5f9ca1d1107cbd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:24:44.388023 19253 tablet_service.cc:1467] Processing CreateTablet for tablet 06fa46925ef2479d8ef219bf37a9e566 (DEFAULT_TABLE table=test-table [id=c69ac2eeb461427aa5f9ca1d1107cbd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:24:44.388965 19384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06fa46925ef2479d8ef219bf37a9e566. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:44.389022 19515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06fa46925ef2479d8ef219bf37a9e566. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:44.389022 19253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06fa46925ef2479d8ef219bf37a9e566. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:44.395761 19598 tablet_bootstrap.cc:492] T 06fa46925ef2479d8ef219bf37a9e566 P 78277fdb6ad44c7ea2d23976408a837c: Bootstrap starting.
I20260926 20:24:44.396909 19600 tablet_bootstrap.cc:492] T 06fa46925ef2479d8ef219bf37a9e566 P 359d3fcb60cc4321be73ebccb93775ec: Bootstrap starting.
I20260926 20:24:44.397229 19599 tablet_bootstrap.cc:492] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166: Bootstrap starting.
I20260926 20:24:44.398034 19598 tablet_bootstrap.cc:654] T 06fa46925ef2479d8ef219bf37a9e566 P 78277fdb6ad44c7ea2d23976408a837c: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:44.398772 19598 log.cc:824] T 06fa46925ef2479d8ef219bf37a9e566 P 78277fdb6ad44c7ea2d23976408a837c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:44.399062 19600 tablet_bootstrap.cc:654] T 06fa46925ef2479d8ef219bf37a9e566 P 359d3fcb60cc4321be73ebccb93775ec: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:44.399055 19599 tablet_bootstrap.cc:654] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:44.399909 19599 log.cc:824] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:44.399909 19600 log.cc:824] T 06fa46925ef2479d8ef219bf37a9e566 P 359d3fcb60cc4321be73ebccb93775ec: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:44.400925 19598 tablet_bootstrap.cc:492] T 06fa46925ef2479d8ef219bf37a9e566 P 78277fdb6ad44c7ea2d23976408a837c: No bootstrap required, opened a new log
I20260926 20:24:44.401124 19598 ts_tablet_manager.cc:1397] T 06fa46925ef2479d8ef219bf37a9e566 P 78277fdb6ad44c7ea2d23976408a837c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:24:44.401834 19599 tablet_bootstrap.cc:492] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166: No bootstrap required, opened a new log
I20260926 20:24:44.402036 19599 ts_tablet_manager.cc:1397] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 20:24:44.402125 19600 tablet_bootstrap.cc:492] T 06fa46925ef2479d8ef219bf37a9e566 P 359d3fcb60cc4321be73ebccb93775ec: No bootstrap required, opened a new log
I20260926 20:24:44.402302 19600 ts_tablet_manager.cc:1397] T 06fa46925ef2479d8ef219bf37a9e566 P 359d3fcb60cc4321be73ebccb93775ec: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:24:44.405210 19599 raft_consensus.cc:348] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359d3fcb60cc4321be73ebccb93775ec" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35663 } } peers { permanent_uuid: "78277fdb6ad44c7ea2d23976408a837c" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38189 } } peers { permanent_uuid: "ffa2f0eb183945419b3689c5e5612166" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 40669 } }
I20260926 20:24:44.405467 19599 raft_consensus.cc:374] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:44.405526 19599 raft_consensus.cc:729] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa2f0eb183945419b3689c5e5612166, State: Initialized, Role: FOLLOWER
I20260926 20:24:44.405540 19598 raft_consensus.cc:348] T 06fa46925ef2479d8ef219bf37a9e566 P 78277fdb6ad44c7ea2d23976408a837c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359d3fcb60cc4321be73ebccb93775ec" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35663 } } peers { permanent_uuid: "78277fdb6ad44c7ea2d23976408a837c" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38189 } } peers { permanent_uuid: "ffa2f0eb183945419b3689c5e5612166" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 40669 } }
I20260926 20:24:44.405787 19598 raft_consensus.cc:374] T 06fa46925ef2479d8ef219bf37a9e566 P 78277fdb6ad44c7ea2d23976408a837c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:44.405849 19598 raft_consensus.cc:729] T 06fa46925ef2479d8ef219bf37a9e566 P 78277fdb6ad44c7ea2d23976408a837c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78277fdb6ad44c7ea2d23976408a837c, State: Initialized, Role: FOLLOWER
I20260926 20:24:44.406005 19599 consensus_queue.cc:260] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359d3fcb60cc4321be73ebccb93775ec" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35663 } } peers { permanent_uuid: "78277fdb6ad44c7ea2d23976408a837c" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38189 } } peers { permanent_uuid: "ffa2f0eb183945419b3689c5e5612166" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 40669 } }
I20260926 20:24:44.406327 19598 consensus_queue.cc:260] T 06fa46925ef2479d8ef219bf37a9e566 P 78277fdb6ad44c7ea2d23976408a837c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359d3fcb60cc4321be73ebccb93775ec" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35663 } } peers { permanent_uuid: "78277fdb6ad44c7ea2d23976408a837c" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38189 } } peers { permanent_uuid: "ffa2f0eb183945419b3689c5e5612166" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 40669 } }
I20260926 20:24:44.406472 19600 raft_consensus.cc:348] T 06fa46925ef2479d8ef219bf37a9e566 P 359d3fcb60cc4321be73ebccb93775ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359d3fcb60cc4321be73ebccb93775ec" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35663 } } peers { permanent_uuid: "78277fdb6ad44c7ea2d23976408a837c" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38189 } } peers { permanent_uuid: "ffa2f0eb183945419b3689c5e5612166" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 40669 } }
I20260926 20:24:44.406715 19600 raft_consensus.cc:374] T 06fa46925ef2479d8ef219bf37a9e566 P 359d3fcb60cc4321be73ebccb93775ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:44.406728 19319 heartbeater.cc:503] Master 127.0.116.254:34181 was elected leader, sending a full tablet report...
I20260926 20:24:44.406772 19600 raft_consensus.cc:729] T 06fa46925ef2479d8ef219bf37a9e566 P 359d3fcb60cc4321be73ebccb93775ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359d3fcb60cc4321be73ebccb93775ec, State: Initialized, Role: FOLLOWER
I20260926 20:24:44.407130 19450 heartbeater.cc:503] Master 127.0.116.254:34181 was elected leader, sending a full tablet report...
I20260926 20:24:44.407225 19600 consensus_queue.cc:260] T 06fa46925ef2479d8ef219bf37a9e566 P 359d3fcb60cc4321be73ebccb93775ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359d3fcb60cc4321be73ebccb93775ec" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35663 } } peers { permanent_uuid: "78277fdb6ad44c7ea2d23976408a837c" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38189 } } peers { permanent_uuid: "ffa2f0eb183945419b3689c5e5612166" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 40669 } }
I20260926 20:24:44.407402 19598 ts_tablet_manager.cc:1428] T 06fa46925ef2479d8ef219bf37a9e566 P 78277fdb6ad44c7ea2d23976408a837c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 20:24:44.408126 19599 ts_tablet_manager.cc:1428] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:24:44.408149 19600 ts_tablet_manager.cc:1428] T 06fa46925ef2479d8ef219bf37a9e566 P 359d3fcb60cc4321be73ebccb93775ec: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:24:44.408438 19581 heartbeater.cc:503] Master 127.0.116.254:34181 was elected leader, sending a full tablet report...
W20260926 20:24:44.451750 19320 tablet.cc:2369] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:24:44.457356 19604 raft_consensus.cc:482] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:24:44.457610 19604 raft_consensus.cc:504] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359d3fcb60cc4321be73ebccb93775ec" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35663 } } peers { permanent_uuid: "78277fdb6ad44c7ea2d23976408a837c" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38189 } } peers { permanent_uuid: "ffa2f0eb183945419b3689c5e5612166" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 40669 } }
I20260926 20:24:44.458875 19604 leader_election.cc:288] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 359d3fcb60cc4321be73ebccb93775ec (127.0.116.195:35663), 78277fdb6ad44c7ea2d23976408a837c (127.0.116.194:38189)
I20260926 20:24:44.463632 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:44.459604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:33263 (local address 127.0.116.194:38189)
0926 20:24:44.459747 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:44.459752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:44.459863 (+ 111us) server_negotiation.cc:407] Connection header received
0926 20:24:44.459966 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:44.459968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:44.460041 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:44.460120 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:44.460366 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.461042 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.462393 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.462999 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.463098 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:44.463143 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:44.463407 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 20:24:44.463478 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:24:44.463632 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:44.459250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:44743 (local address 127.0.116.195:35663)
0926 20:24:44.459415 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:44.459422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:44.459439 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:24:44.459517 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:44.459521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:44.459572 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:44.459645 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:44.460036 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.460812 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.462299 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.462929 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.463068 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:44.463121 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:44.463413 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 20:24:44.463478 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:24:44.463647 19321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:44.459067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:35663 (local address 127.0.116.193:44743)
0926 20:24:44.459230 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:44.459247 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:44.459346 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:44.459761 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:44.459764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:44.459784 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:44.459906 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:44.459916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.460921 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.460923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:44.462165 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:44.462176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.463078 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.463083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:44.463252 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:44.463303 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 20:24:44.463375 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 20:24:44.463467 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":73}
I20260926 20:24:44.463655 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:44.459444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:38189 (local address 127.0.116.193:33263)
0926 20:24:44.459778 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:44.459791 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:44.459876 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:44.460146 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:44.460148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:44.460163 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:44.460259 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:44.460264 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.461145 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.461148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:44.462276 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:44.462286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.463099 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.463102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:44.463255 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:44.463296 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 20:24:44.463375 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 20:24:44.463467 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":51,"thread_start_us":145,"threads_started":1}
I20260926 20:24:44.464329 19535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06fa46925ef2479d8ef219bf37a9e566" candidate_uuid: "ffa2f0eb183945419b3689c5e5612166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359d3fcb60cc4321be73ebccb93775ec" is_pre_election: true
I20260926 20:24:44.464331 19404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06fa46925ef2479d8ef219bf37a9e566" candidate_uuid: "ffa2f0eb183945419b3689c5e5612166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78277fdb6ad44c7ea2d23976408a837c" is_pre_election: true
I20260926 20:24:44.464649 19535 raft_consensus.cc:2454] T 06fa46925ef2479d8ef219bf37a9e566 P 359d3fcb60cc4321be73ebccb93775ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffa2f0eb183945419b3689c5e5612166 in term 0.
I20260926 20:24:44.464658 19404 raft_consensus.cc:2454] T 06fa46925ef2479d8ef219bf37a9e566 P 78277fdb6ad44c7ea2d23976408a837c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffa2f0eb183945419b3689c5e5612166 in term 0.
I20260926 20:24:44.465193 19208 leader_election.cc:302] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78277fdb6ad44c7ea2d23976408a837c, ffa2f0eb183945419b3689c5e5612166; no voters:
I20260926 20:24:44.465473 19604 raft_consensus.cc:2781] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:24:44.465550 19604 raft_consensus.cc:482] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:24:44.465581 19604 raft_consensus.cc:3037] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:44.466645 19604 raft_consensus.cc:504] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359d3fcb60cc4321be73ebccb93775ec" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35663 } } peers { permanent_uuid: "78277fdb6ad44c7ea2d23976408a837c" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38189 } } peers { permanent_uuid: "ffa2f0eb183945419b3689c5e5612166" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 40669 } }
I20260926 20:24:44.466995 19604 leader_election.cc:288] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166 [CANDIDATE]: Term 1 election: Requested vote from peers 359d3fcb60cc4321be73ebccb93775ec (127.0.116.195:35663), 78277fdb6ad44c7ea2d23976408a837c (127.0.116.194:38189)
I20260926 20:24:44.467376 19535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06fa46925ef2479d8ef219bf37a9e566" candidate_uuid: "ffa2f0eb183945419b3689c5e5612166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359d3fcb60cc4321be73ebccb93775ec"
I20260926 20:24:44.467466 19404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06fa46925ef2479d8ef219bf37a9e566" candidate_uuid: "ffa2f0eb183945419b3689c5e5612166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78277fdb6ad44c7ea2d23976408a837c"
I20260926 20:24:44.467484 19535 raft_consensus.cc:3037] T 06fa46925ef2479d8ef219bf37a9e566 P 359d3fcb60cc4321be73ebccb93775ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:44.467639 19404 raft_consensus.cc:3037] T 06fa46925ef2479d8ef219bf37a9e566 P 78277fdb6ad44c7ea2d23976408a837c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:44.468564 19535 raft_consensus.cc:2454] T 06fa46925ef2479d8ef219bf37a9e566 P 359d3fcb60cc4321be73ebccb93775ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffa2f0eb183945419b3689c5e5612166 in term 1.
I20260926 20:24:44.468886 19208 leader_election.cc:302] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359d3fcb60cc4321be73ebccb93775ec, ffa2f0eb183945419b3689c5e5612166; no voters:
I20260926 20:24:44.468878 19404 raft_consensus.cc:2454] T 06fa46925ef2479d8ef219bf37a9e566 P 78277fdb6ad44c7ea2d23976408a837c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffa2f0eb183945419b3689c5e5612166 in term 1.
I20260926 20:24:44.469038 19604 raft_consensus.cc:2781] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:24:44.469302 19604 raft_consensus.cc:686] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166 [term 1 LEADER]: Becoming Leader. State: Replica: ffa2f0eb183945419b3689c5e5612166, State: Running, Role: LEADER
I20260926 20:24:44.469630 19604 consensus_queue.cc:237] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359d3fcb60cc4321be73ebccb93775ec" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35663 } } peers { permanent_uuid: "78277fdb6ad44c7ea2d23976408a837c" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38189 } } peers { permanent_uuid: "ffa2f0eb183945419b3689c5e5612166" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 40669 } }
I20260926 20:24:44.472695 19131 catalog_manager.cc:5495] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffa2f0eb183945419b3689c5e5612166 (127.0.116.193). New cstate: current_term: 1 leader_uuid: "ffa2f0eb183945419b3689c5e5612166" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359d3fcb60cc4321be73ebccb93775ec" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78277fdb6ad44c7ea2d23976408a837c" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffa2f0eb183945419b3689c5e5612166" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 40669 } health_report { overall_health: HEALTHY } } }
I20260926 20:24:44.484164 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:44.480491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49156 (local address 127.0.116.193:40669)
0926 20:24:44.480613 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:44.480616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:44.480629 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:24:44.480696 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:44.480698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:44.480748 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:44.480841 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:24:44.481075 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.481573 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.482456 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:44.482671 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:44.482757 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:44.483158 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:24:44.483683 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:24:44.483848 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:44.483959 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 20:24:44.484010 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:24:44.490252 19404 raft_consensus.cc:1261] T 06fa46925ef2479d8ef219bf37a9e566 P 78277fdb6ad44c7ea2d23976408a837c [term 1 FOLLOWER]: Refusing update from remote peer ffa2f0eb183945419b3689c5e5612166: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:24:44.490252 19535 raft_consensus.cc:1261] T 06fa46925ef2479d8ef219bf37a9e566 P 359d3fcb60cc4321be73ebccb93775ec [term 1 FOLLOWER]: Refusing update from remote peer ffa2f0eb183945419b3689c5e5612166: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:24:44.491016 19608 consensus_queue.cc:1035] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78277fdb6ad44c7ea2d23976408a837c" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 38189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:24:44.491218 19604 consensus_queue.cc:1035] T 06fa46925ef2479d8ef219bf37a9e566 P ffa2f0eb183945419b3689c5e5612166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359d3fcb60cc4321be73ebccb93775ec" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 35663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:24:44.500792 19613 mvcc.cc:205] Tried to move back new op lower bound from 7333700749264089088 to 7333700749190115328. Current Snapshot: MvccSnapshot[applied={T|T < 7333700749264089088}]
I20260926 20:24:44.508280 19131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:24:44.508455 19131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:24:44.510815 19131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 066979cdee9f47b0a91c67944d3cca56: Sending DeleteTablet for 3 replicas of tablet 06fa46925ef2479d8ef219bf37a9e566
I20260926 20:24:44.511699 19253 tablet_service.cc:1514] Processing DeleteTablet for tablet 06fa46925ef2479d8ef219bf37a9e566 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:24:44 UTC) from {username='slave'} at 127.0.0.1:49142
I20260926 20:24:44.511926 19515 tablet_service.cc:1514] Processing DeleteTablet for tablet 06fa46925ef2479d8ef219bf37a9e566 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:24:44 UTC) from {username='slave'} at 127.0.0.1:47776
I20260926 20:24:44.512321 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 19191
I20260926 20:24:44.512274 19384 tablet_service.cc:1514] Processing DeleteTablet for tablet 06fa46925ef2479d8ef219bf37a9e566 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:24:44 UTC) from {username='slave'} at 127.0.0.1:58206
I20260926 20:24:44.513494 19623 tablet_replica.cc:330] T 06fa46925ef2479d8ef219bf37a9e566 P 78277fdb6ad44c7ea2d23976408a837c: stopping tablet replica
I20260926 20:24:44.513854 19623 raft_consensus.cc:2229] T 06fa46925ef2479d8ef219bf37a9e566 P 78277fdb6ad44c7ea2d23976408a837c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:24:44.514205 19623 raft_consensus.cc:2258] T 06fa46925ef2479d8ef219bf37a9e566 P 78277fdb6ad44c7ea2d23976408a837c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:24:44.514930 19622 tablet_replica.cc:330] T 06fa46925ef2479d8ef219bf37a9e566 P 359d3fcb60cc4321be73ebccb93775ec: stopping tablet replica
I20260926 20:24:44.515234 19622 raft_consensus.cc:2229] T 06fa46925ef2479d8ef219bf37a9e566 P 359d3fcb60cc4321be73ebccb93775ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:24:44.515594 19622 raft_consensus.cc:2258] T 06fa46925ef2479d8ef219bf37a9e566 P 359d3fcb60cc4321be73ebccb93775ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:24:44.516237 19623 ts_tablet_manager.cc:1905] T 06fa46925ef2479d8ef219bf37a9e566 P 78277fdb6ad44c7ea2d23976408a837c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:24:44.517088 19622 ts_tablet_manager.cc:1905] T 06fa46925ef2479d8ef219bf37a9e566 P 359d3fcb60cc4321be73ebccb93775ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:24:44.519131 19623 ts_tablet_manager.cc:1918] T 06fa46925ef2479d8ef219bf37a9e566 P 78277fdb6ad44c7ea2d23976408a837c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:24:44.519227 19623 log.cc:1196] T 06fa46925ef2479d8ef219bf37a9e566 P 78277fdb6ad44c7ea2d23976408a837c: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/06fa46925ef2479d8ef219bf37a9e566
I20260926 20:24:44.519583 19623 ts_tablet_manager.cc:1939] T 06fa46925ef2479d8ef219bf37a9e566 P 78277fdb6ad44c7ea2d23976408a837c: Deleting consensus metadata
W20260926 20:24:44.519783 19116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.116.193:40669 (error 108)
I20260926 20:24:44.520016 19622 ts_tablet_manager.cc:1918] T 06fa46925ef2479d8ef219bf37a9e566 P 359d3fcb60cc4321be73ebccb93775ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:24:44.520105 19622 log.cc:1196] T 06fa46925ef2479d8ef219bf37a9e566 P 359d3fcb60cc4321be73ebccb93775ec: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/06fa46925ef2479d8ef219bf37a9e566
I20260926 20:24:44.520401 19622 ts_tablet_manager.cc:1939] T 06fa46925ef2479d8ef219bf37a9e566 P 359d3fcb60cc4321be73ebccb93775ec: Deleting consensus metadata
I20260926 20:24:44.520648 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 19322
I20260926 20:24:44.521086 19595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:44.520672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:40669 (local address 127.0.0.1:49168)
0926 20:24:44.520889 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:44.520968 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.193:40669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260926 20:24:44.521438 19116 catalog_manager.cc:4584] TS ffa2f0eb183945419b3689c5e5612166 (127.0.116.193:40669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 06fa46925ef2479d8ef219bf37a9e566: Network error: Client connection negotiation failed: client connection to 127.0.116.193:40669: connect: Connection refused (error 111)
I20260926 20:24:44.521957 19117 catalog_manager.cc:4841] TS 359d3fcb60cc4321be73ebccb93775ec (127.0.116.195:35663): tablet 06fa46925ef2479d8ef219bf37a9e566 (table test-table [id=c69ac2eeb461427aa5f9ca1d1107cbd6]) successfully deleted
I20260926 20:24:44.527263 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 19453
I20260926 20:24:44.527283 19595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:44.526930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:38189 (local address 127.0.0.1:58214)
0926 20:24:44.527108 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:44.527193 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.194:38189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260926 20:24:44.532824 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 19100
2026-09-26T20:24:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1033 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:24:44 dist-test-slave-qsm9 krb5kdc[19627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:24:44 dist-test-slave-qsm9 krb5kdc[19627](info): set up 2 sockets
Sep 26 20:24:44 dist-test-slave-qsm9 krb5kdc[19627](info): commencing operation
krb5kdc: starting...
W20260926 20:24:45.591813 467 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:24:45 dist-test-slave-qsm9 krb5kdc[19627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:24:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:24:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:24:45.723464 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:44185
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:44185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:24:45.836329 19640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:45.836616 19640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:45.836658 19640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:45.840843 19640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:24:45.840914 19640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:45.840935 19640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:45.840952 19640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:24:45.840967 19640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:24:45.846293 19640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:44185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:44185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:45.847411 19640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:45.848495 19640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:45.857388 19645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:45.857388 19646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:45.857637 19640 server_base.cc:1034] running on GCE node
W20260926 20:24:45.857412 19648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:45.858421 19640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:45.859604 19640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:45.860805 19640 hybrid_clock.cc:648] HybridClock initialized: now 1790454285860787 us; error 33 us; skew 500 ppm
Sep 26 20:24:45 dist-test-slave-qsm9 krb5kdc[19627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:45.863804 19640 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:24:45.865065 19640 webserver.cc:460] Webserver started at http://127.0.116.254:34333/ using document root <none> and password file <none>
I20260926 20:24:45.865808 19640 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:45.865899 19640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:45.866145 19640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:45.868939 19640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "656f594ed94a460f88dcb2c8470f492f"
format_stamp: "Formatted at 2026-09-26 20:24:45 on dist-test-slave-qsm9"
server_key: "7cbe8c3b00d937a99ac1c6c0adcd83c2"
server_key_iv: "a07000c8d0dafa59ce75989ebbce23be"
server_key_version: "encryptionkey@0"
I20260926 20:24:45.869580 19640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "656f594ed94a460f88dcb2c8470f492f"
format_stamp: "Formatted at 2026-09-26 20:24:45 on dist-test-slave-qsm9"
server_key: "7cbe8c3b00d937a99ac1c6c0adcd83c2"
server_key_iv: "a07000c8d0dafa59ce75989ebbce23be"
server_key_version: "encryptionkey@0"
I20260926 20:24:45.874269 19640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:24:45.877123 19655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:45.878458 19640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 20:24:45.878613 19640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "656f594ed94a460f88dcb2c8470f492f"
format_stamp: "Formatted at 2026-09-26 20:24:45 on dist-test-slave-qsm9"
server_key: "7cbe8c3b00d937a99ac1c6c0adcd83c2"
server_key_iv: "a07000c8d0dafa59ce75989ebbce23be"
server_key_version: "encryptionkey@0"
I20260926 20:24:45.878738 19640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:45.913182 19640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:45.916800 19640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:45.917057 19640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:45.924877 19640 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:44185
I20260926 20:24:45.924896 19707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:44185 every 8 connection(s)
I20260926 20:24:45.925907 19640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:24:45.928741 19708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:45.934253 19708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f: Bootstrap starting.
I20260926 20:24:45.934598 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 19640
I20260926 20:24:45.934729 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:24:45.935019 467 external_mini_cluster.cc:1428] Setting key 5694a6112af31d83b0ebecea87e7a9e8
I20260926 20:24:45.937095 19708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:45.937825 19708 log.cc:824] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:45.939728 19708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f: No bootstrap required, opened a new log
I20260926 20:24:45.943029 19708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f594ed94a460f88dcb2c8470f492f" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44185 } }
I20260926 20:24:45.943277 19708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:45.943341 19708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 656f594ed94a460f88dcb2c8470f492f, State: Initialized, Role: FOLLOWER
Sep 26 20:24:45 dist-test-slave-qsm9 krb5kdc[19627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:24:45.943861 19708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f594ed94a460f88dcb2c8470f492f" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44185 } }
I20260926 20:24:45.944021 19708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:24:45.944077 19708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:24:45.944150 19708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:45.945000 19708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f594ed94a460f88dcb2c8470f492f" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44185 } }
I20260926 20:24:45.945333 19708 leader_election.cc:302] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 656f594ed94a460f88dcb2c8470f492f; no voters:
I20260926 20:24:45.945639 19708 leader_election.cc:288] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:24:45.945756 19713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:24:45.946039 19713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f [term 1 LEADER]: Becoming Leader. State: Replica: 656f594ed94a460f88dcb2c8470f492f, State: Running, Role: LEADER
I20260926 20:24:45.946332 19713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f594ed94a460f88dcb2c8470f492f" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44185 } }
I20260926 20:24:45.947033 19708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:24:45.948011 19714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "656f594ed94a460f88dcb2c8470f492f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f594ed94a460f88dcb2c8470f492f" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44185 } } }
I20260926 20:24:45.948132 19714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:45.948378 19715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 656f594ed94a460f88dcb2c8470f492f. Latest consensus state: current_term: 1 leader_uuid: "656f594ed94a460f88dcb2c8470f492f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656f594ed94a460f88dcb2c8470f492f" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 44185 } } }
I20260926 20:24:45.948490 19715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:45.948753 19722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:24:45.950671 19711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:45.936476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56278 (local address 127.0.116.254:44185)
0926 20:24:45.936920 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:45.936938 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:45.936977 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:24:45.937667 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:45.937685 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:45.938062 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:45.938390 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:45.939216 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:45.940299 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:45.941161 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:45.941638 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:45.941811 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:45.944675 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:45.944680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:45.944693 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:45.944718 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:45.946829 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:45.947337 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:45.947344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:45.947352 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:45.947532 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:45.947872 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:45.947878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:45.947882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:45.948446 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:45.948664 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:45.949411 (+ 747us) server_negotiation.cc:299] Negotiation successful
0926 20:24:45.949707 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":135,"threads_started":1}
I20260926 20:24:45.952033 19722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:24:45.958076 19722 catalog_manager.cc:1347] Generated new cluster ID: dab55b04026b44e1b1080d5ac6bbc891
I20260926 20:24:45.958164 19722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:24:45.979285 19722 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:24:45.980643 19722 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:24:45.989308 19722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f: Generated new TSK 0
I20260926 20:24:45.990164 19722 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
Entry for principal kudu/127.0.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab.
I20260926 20:24:46.028892 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44185
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:24:46.145498 19734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:46.145794 19734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:46.145841 19734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:46.150381 19734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:46.150465 19734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:46.150559 19734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:24:46.156396 19734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/kudu_127.0.116.193.keytab
--principal=kudu/127.0.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:46.157584 19734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:46.158641 19734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:46.168848 19742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:46.168887 19740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:46.169104 19734 server_base.cc:1034] running on GCE node
W20260926 20:24:46.169224 19739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:46.169687 19734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:46.170312 19734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:46.171484 19734 hybrid_clock.cc:648] HybridClock initialized: now 1790454286171449 us; error 48 us; skew 500 ppm
Sep 26 20:24:46 dist-test-slave-qsm9 krb5kdc[19627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:46.174357 19734 init.cc:377] Logged in from keytab as kudu/127.0.116.193@KRBTEST.COM (short username kudu)
I20260926 20:24:46.175283 19734 webserver.cc:460] Webserver started at http://127.0.116.193:38973/ using document root <none> and password file <none>
I20260926 20:24:46.175899 19734 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:46.175961 19734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:46.176127 19734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:46.178162 19734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "7973b491457a4e81a3bf9a8edcef52d6"
format_stamp: "Formatted at 2026-09-26 20:24:46 on dist-test-slave-qsm9"
server_key: "0418ab27901d8053ea08694f1b8d0105"
server_key_iv: "898f7cc564632c01447b8ba17da3553a"
server_key_version: "encryptionkey@0"
I20260926 20:24:46.178692 19734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "7973b491457a4e81a3bf9a8edcef52d6"
format_stamp: "Formatted at 2026-09-26 20:24:46 on dist-test-slave-qsm9"
server_key: "0418ab27901d8053ea08694f1b8d0105"
server_key_iv: "898f7cc564632c01447b8ba17da3553a"
server_key_version: "encryptionkey@0"
I20260926 20:24:46.182850 19734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:24:46.185582 19749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:46.186879 19734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:24:46.187083 19734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "7973b491457a4e81a3bf9a8edcef52d6"
format_stamp: "Formatted at 2026-09-26 20:24:46 on dist-test-slave-qsm9"
server_key: "0418ab27901d8053ea08694f1b8d0105"
server_key_iv: "898f7cc564632c01447b8ba17da3553a"
server_key_version: "encryptionkey@0"
I20260926 20:24:46.187220 19734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:46.205736 19734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:46.209295 19734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:46.209546 19734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:46.210223 19734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:46.211185 19734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:24:46.211266 19734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:46.211350 19734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:24:46.211403 19734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:46.221421 19734 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:32817
I20260926 20:24:46.221446 19862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:32817 every 8 connection(s)
I20260926 20:24:46.222577 19734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:24:46.226138 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 19734
I20260926 20:24:46.226303 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:24:46.226612 467 external_mini_cluster.cc:1428] Setting key 2e32810dba37aa79c022436531a72b2f
Sep 26 20:24:46 dist-test-slave-qsm9 krb5kdc[19627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.193@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.116.194@KRBTEST.COM; defaulting to no policy
I20260926 20:24:46.239312 19711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:46.224578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:45621 (local address 127.0.116.254:44185)
0926 20:24:46.224717 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:46.224721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:46.225588 (+ 867us) server_negotiation.cc:407] Connection header received
0926 20:24:46.227202 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:46.227207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:46.227268 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:46.227338 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:46.229194 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.229726 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.231122 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.231603 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.231727 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:46.235035 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:46.235042 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:46.235048 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:46.235092 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:46.237185 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:46.237719 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:46.237725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:46.237729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:46.237843 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:46.238219 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:46.238222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:46.238223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:46.238424 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:46.238512 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:46.239053 (+ 541us) server_negotiation.cc:299] Negotiation successful
0926 20:24:46.239175 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:24:46.240185 19865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:46.224909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44185 (local address 127.0.116.193:45621)
0926 20:24:46.225410 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:46.225452 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:46.226846 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:46.227645 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:46.227653 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:46.228234 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:46.228998 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:46.229026 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.229912 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.229917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:46.230978 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:46.230989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.231757 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.231760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:46.231919 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:46.232697 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:24:46.232725 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:46.234829 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:46.237325 (+ 2496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:46.237330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:46.237344 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:46.237603 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:46.237973 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:46.237976 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:46.237977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:46.238123 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:46.238521 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:46.238527 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:46.238778 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 20:24:46.238964 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 20:24:46.239240 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.194@KRBTEST.COM" created.
I20260926 20:24:46.241583 19863 heartbeater.cc:347] Connected to a master server at 127.0.116.254:44185
I20260926 20:24:46.241850 19863 heartbeater.cc:464] Registering TS with master...
I20260926 20:24:46.242355 19863 heartbeater.cc:511] Master 127.0.116.254:44185 requested a full tablet report, sending...
I20260926 20:24:46.244035 19672 ts_manager.cc:194] Registered new tserver with Master: 7973b491457a4e81a3bf9a8edcef52d6 (127.0.116.193:32817)
I20260926 20:24:46.245174 19672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.193@KRBTEST.COM'} at 127.0.116.193:45621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
Entry for principal kudu/127.0.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab.
I20260926 20:24:46.258653 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44185
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:24:46.372943 19868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:46.373219 19868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:46.373261 19868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:46.377270 19868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:46.377327 19868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:46.377385 19868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:24:46.382850 19868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/kudu_127.0.116.194.keytab
--principal=kudu/127.0.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:46.384210 19868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:46.385236 19868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:46.395453 19876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:46.395458 19874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:46.395458 19873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:46.395947 19868 server_base.cc:1034] running on GCE node
I20260926 20:24:46.396482 19868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:46.397212 19868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:46.398418 19868 hybrid_clock.cc:648] HybridClock initialized: now 1790454286398386 us; error 51 us; skew 500 ppm
Sep 26 20:24:46 dist-test-slave-qsm9 krb5kdc[19627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:46.401588 19868 init.cc:377] Logged in from keytab as kudu/127.0.116.194@KRBTEST.COM (short username kudu)
I20260926 20:24:46.402516 19868 webserver.cc:460] Webserver started at http://127.0.116.194:45547/ using document root <none> and password file <none>
I20260926 20:24:46.403147 19868 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:46.403241 19868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:46.403478 19868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:46.405684 19868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "0f98711eaab94022940a3565f3a598c7"
format_stamp: "Formatted at 2026-09-26 20:24:46 on dist-test-slave-qsm9"
server_key: "eea520a81b9a6c36bb68d569ba97e1dd"
server_key_iv: "1971c8b8a7c75ea4d2527ee22de6d0fe"
server_key_version: "encryptionkey@0"
I20260926 20:24:46.406266 19868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "0f98711eaab94022940a3565f3a598c7"
format_stamp: "Formatted at 2026-09-26 20:24:46 on dist-test-slave-qsm9"
server_key: "eea520a81b9a6c36bb68d569ba97e1dd"
server_key_iv: "1971c8b8a7c75ea4d2527ee22de6d0fe"
server_key_version: "encryptionkey@0"
I20260926 20:24:46.410269 19868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 20:24:46.412992 19883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:46.414350 19868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:24:46.414513 19868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "0f98711eaab94022940a3565f3a598c7"
format_stamp: "Formatted at 2026-09-26 20:24:46 on dist-test-slave-qsm9"
server_key: "eea520a81b9a6c36bb68d569ba97e1dd"
server_key_iv: "1971c8b8a7c75ea4d2527ee22de6d0fe"
server_key_version: "encryptionkey@0"
I20260926 20:24:46.414642 19868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:46.437399 19868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:46.441399 19868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:46.441650 19868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:46.442338 19868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:46.443271 19868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:24:46.443362 19868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:46.443446 19868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:24:46.443527 19868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:46.454394 19868 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:44455
I20260926 20:24:46.454452 19996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:44455 every 8 connection(s)
I20260926 20:24:46.455420 19868 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:24:46.456197 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 19868
I20260926 20:24:46.456343 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:24:46.456626 467 external_mini_cluster.cc:1428] Setting key c48f0a8231b0461c9142ff4390bdcbf7
Sep 26 20:24:46 dist-test-slave-qsm9 krb5kdc[19627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.194@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.116.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.195@KRBTEST.COM" created.
I20260926 20:24:46.470404 19711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:46.457367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:56539 (local address 127.0.116.254:44185)
0926 20:24:46.457485 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:46.457492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:46.458632 (+ 1140us) server_negotiation.cc:407] Connection header received
0926 20:24:46.460003 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:46.460006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:46.460061 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:46.460169 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:46.461358 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.461892 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.463075 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.463309 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.463432 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:46.466414 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:46.466417 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:46.466419 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:46.466446 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:46.468033 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:46.468562 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:46.468566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:46.468567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:46.468654 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:46.469009 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:46.469013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:46.469014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:46.469355 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:46.469454 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:46.470075 (+ 621us) server_negotiation.cc:299] Negotiation successful
0926 20:24:46.470218 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 20:24:46.471264 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:46.457976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44185 (local address 127.0.116.194:56539)
0926 20:24:46.458459 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:46.458512 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:46.459784 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:46.460346 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:46.460354 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:46.460717 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:46.461212 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:46.461226 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.462024 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.462032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:46.462947 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:46.462956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.463473 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.463478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:46.463674 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:46.464362 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:24:46.464381 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:46.466182 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:46.468150 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:46.468156 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:46.468169 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:46.468441 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:46.468747 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:46.468749 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:46.468750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:46.468883 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:46.469485 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:46.469493 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:46.469773 (+ 280us) client_negotiation.cc:769] Sending connection context
0926 20:24:46.470012 (+ 239us) client_negotiation.cc:240] Negotiation successful
0926 20:24:46.470393 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":123,"threads_started":1}
I20260926 20:24:46.472692 19997 heartbeater.cc:347] Connected to a master server at 127.0.116.254:44185
I20260926 20:24:46.472965 19997 heartbeater.cc:464] Registering TS with master...
I20260926 20:24:46.473515 19997 heartbeater.cc:511] Master 127.0.116.254:44185 requested a full tablet report, sending...
I20260926 20:24:46.474798 19672 ts_manager.cc:194] Registered new tserver with Master: 0f98711eaab94022940a3565f3a598c7 (127.0.116.194:44455)
I20260926 20:24:46.475478 19672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.194@KRBTEST.COM'} at 127.0.116.194:56539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
Entry for principal kudu/127.0.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab.
I20260926 20:24:46.487473 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44185
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:24:46.601181 20002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:46.601536 20002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:46.601631 20002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:46.606626 20002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:46.606753 20002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:46.606859 20002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:24:46.612644 20002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/krb5kdc/kudu_127.0.116.195.keytab
--principal=kudu/127.0.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:44185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:46.613802 20002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:46.614902 20002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:46.624963 20007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:46.624945 20010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:46.624950 20008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:46.625178 20002 server_base.cc:1034] running on GCE node
I20260926 20:24:46.625662 20002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:46.626261 20002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:46.627451 20002 hybrid_clock.cc:648] HybridClock initialized: now 1790454286627431 us; error 32 us; skew 500 ppm
Sep 26 20:24:46 dist-test-slave-qsm9 krb5kdc[19627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:46.630337 20002 init.cc:377] Logged in from keytab as kudu/127.0.116.195@KRBTEST.COM (short username kudu)
I20260926 20:24:46.631313 20002 webserver.cc:460] Webserver started at http://127.0.116.195:43273/ using document root <none> and password file <none>
I20260926 20:24:46.631907 20002 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:46.632001 20002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:46.632222 20002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:46.634245 20002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "9dbf6379a4394afe981f332f62c56b87"
format_stamp: "Formatted at 2026-09-26 20:24:46 on dist-test-slave-qsm9"
server_key: "0e9977e94f4591cd4a5322923847f278"
server_key_iv: "9dd2645261ca8c8702b0f4d6dd520f05"
server_key_version: "encryptionkey@0"
I20260926 20:24:46.634807 20002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "9dbf6379a4394afe981f332f62c56b87"
format_stamp: "Formatted at 2026-09-26 20:24:46 on dist-test-slave-qsm9"
server_key: "0e9977e94f4591cd4a5322923847f278"
server_key_iv: "9dd2645261ca8c8702b0f4d6dd520f05"
server_key_version: "encryptionkey@0"
I20260926 20:24:46.638491 20002 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 20:24:46.641055 20017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:46.642251 20002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:24:46.642382 20002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "9dbf6379a4394afe981f332f62c56b87"
format_stamp: "Formatted at 2026-09-26 20:24:46 on dist-test-slave-qsm9"
server_key: "0e9977e94f4591cd4a5322923847f278"
server_key_iv: "9dd2645261ca8c8702b0f4d6dd520f05"
server_key_version: "encryptionkey@0"
I20260926 20:24:46.642513 20002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:46.664911 20002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:46.668823 20002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:46.669054 20002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:46.669713 20002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:46.670619 20002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:24:46.670693 20002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:46.670779 20002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:24:46.670827 20002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:46.681357 20002 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:34907
I20260926 20:24:46.681380 20130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:34907 every 8 connection(s)
I20260926 20:24:46.682340 20002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:24:46.684860 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 20002
I20260926 20:24:46.685004 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:24:46.685381 467 external_mini_cluster.cc:1428] Setting key 24b35dc3656fbbe7607908b8126dd852
Sep 26 20:24:46 dist-test-slave-qsm9 krb5kdc[19627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.195@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:24:46.696622 19711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:46.684438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:44995 (local address 127.0.116.254:44185)
0926 20:24:46.684587 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:46.684591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:46.685643 (+ 1052us) server_negotiation.cc:407] Connection header received
0926 20:24:46.686992 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:46.686996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:46.687052 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:46.687150 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:46.688458 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.688999 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.690125 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.690345 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.690447 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:46.693195 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:46.693198 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:46.693200 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:46.693223 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:46.694558 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:46.695051 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:46.695054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:46.695055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:46.695148 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:46.695580 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:46.695583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:46.695584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:46.695803 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:46.695899 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:46.696380 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 20:24:46.696493 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:24:46.697474 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:46.684868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:44185 (local address 127.0.116.195:44995)
0926 20:24:46.685463 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:46.685526 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:46.686796 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:46.687293 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:46.687299 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:46.687807 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:46.688283 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:46.688297 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.689108 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.689111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:46.690016 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:46.690025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.690503 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.690506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:46.690748 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:46.691398 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:24:46.691416 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:46.693028 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:46.694667 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:46.694672 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:46.694685 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:46.694943 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:46.695272 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:24:46.695275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:24:46.695277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:24:46.695419 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:24:46.695905 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:46.695911 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:46.696154 (+ 243us) client_negotiation.cc:769] Sending connection context
0926 20:24:46.696319 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 20:24:46.696579 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":139,"threads_started":1}
I20260926 20:24:46.699012 20131 heartbeater.cc:347] Connected to a master server at 127.0.116.254:44185
I20260926 20:24:46.699265 20131 heartbeater.cc:464] Registering TS with master...
I20260926 20:24:46.699848 20131 heartbeater.cc:511] Master 127.0.116.254:44185 requested a full tablet report, sending...
I20260926 20:24:46.700868 19671 ts_manager.cc:194] Registered new tserver with Master: 9dbf6379a4394afe981f332f62c56b87 (127.0.116.195:34907)
I20260926 20:24:46.701460 19671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.116.195@KRBTEST.COM'} at 127.0.116.195:44995
I20260926 20:24:46.711783 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:24:46.722034 19711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:46.713890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56290 (local address 127.0.116.254:44185)
0926 20:24:46.714052 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:46.714057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:46.714229 (+ 172us) server_negotiation.cc:407] Connection header received
0926 20:24:46.714401 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:46.714404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:46.714456 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:46.714536 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:46.715278 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.715893 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.716753 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.716981 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.717083 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:46.718572 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:46.718576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:46.718578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:46.718615 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:46.720317 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:46.720760 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:46.720764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:46.720766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:46.720855 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:46.721124 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:46.721128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:46.721130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:46.721366 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:46.721512 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:46.721745 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 20:24:46.721861 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:24:46.725116 19671 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:24:46.728219 19671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:24:46.743577 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:46.737479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36238 (local address 127.0.116.195:34907)
0926 20:24:46.737989 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:46.738005 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:46.738032 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:24:46.738116 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:46.738122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:46.738353 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:46.738539 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:46.739091 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.740347 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.741975 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.742826 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.742945 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:46.743096 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:46.743285 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 20:24:46.743402 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":187,"threads_started":1}
I20260926 20:24:46.743635 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:46.738139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:32817 (local address 127.0.0.1:33230)
0926 20:24:46.738532 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:46.738547 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:46.738636 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:46.739250 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:46.739253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:46.739271 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:46.739382 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:46.739389 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.740908 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.740912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:46.741855 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:46.741865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.743182 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.743185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:46.743309 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:46.743333 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:24:46.743402 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:24:46.743476 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":15,"thread_start_us":97,"threads_started":1}
I20260926 20:24:46.743606 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:46.737284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:34907 (local address 127.0.0.1:36238)
0926 20:24:46.737705 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:46.737771 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:46.737900 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:46.738758 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:46.738764 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:46.738790 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:46.738956 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:46.738964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.740522 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.740525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:46.741820 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:46.741829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.742950 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.742953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:46.743074 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:46.743116 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 20:24:46.743237 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 20:24:46.743364 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":56,"thread_start_us":145,"threads_started":1}
I20260926 20:24:46.744292 20146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:46.738493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:44455 (local address 127.0.0.1:34226)
0926 20:24:46.738813 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:46.738827 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:46.738902 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:46.739728 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:46.739731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:46.739749 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:46.739851 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:46.739858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.740968 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.740971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:46.742568 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:46.742575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.743862 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.743866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:46.743994 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:46.744018 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:24:46.744087 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:24:46.744151 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":29,"thread_start_us":103,"threads_started":1}
I20260926 20:24:46.744709 20145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:46.738352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33230 (local address 127.0.116.193:32817)
0926 20:24:46.738755 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:46.738769 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:46.738791 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:24:46.738871 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:46.738876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:46.739099 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:46.739574 (+ 475us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:46.739721 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.740765 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.741978 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.742826 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.742940 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:46.743096 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:46.744282 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0926 20:24:46.744579 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":93,"threads_started":1}
I20260926 20:24:46.744719 20147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:46.738963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34226 (local address 127.0.116.194:44455)
0926 20:24:46.739302 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:46.739316 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:46.739334 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:24:46.739400 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:46.739404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:46.739594 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:46.739742 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:46.739957 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.740820 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.742686 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.743461 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.743648 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:46.743808 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:46.744459 (+ 651us) server_negotiation.cc:299] Negotiation successful
0926 20:24:46.744562 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":150,"threads_started":1}
I20260926 20:24:46.747256 19931 tablet_service.cc:1467] Processing CreateTablet for tablet 61280d4839ff4f9bb9a3f92ad70fbc0c (DEFAULT_TABLE table=test-table [id=6b5c054a75e04b5fa4542d989aa0fd6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:24:46.747260 20065 tablet_service.cc:1467] Processing CreateTablet for tablet 61280d4839ff4f9bb9a3f92ad70fbc0c (DEFAULT_TABLE table=test-table [id=6b5c054a75e04b5fa4542d989aa0fd6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:24:46.747258 19797 tablet_service.cc:1467] Processing CreateTablet for tablet 61280d4839ff4f9bb9a3f92ad70fbc0c (DEFAULT_TABLE table=test-table [id=6b5c054a75e04b5fa4542d989aa0fd6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:24:46.748431 19797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61280d4839ff4f9bb9a3f92ad70fbc0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:46.748431 20065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61280d4839ff4f9bb9a3f92ad70fbc0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:46.749677 19931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61280d4839ff4f9bb9a3f92ad70fbc0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:46.755949 20148 tablet_bootstrap.cc:492] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 0f98711eaab94022940a3565f3a598c7: Bootstrap starting.
I20260926 20:24:46.756852 20150 tablet_bootstrap.cc:492] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87: Bootstrap starting.
I20260926 20:24:46.758088 20148 tablet_bootstrap.cc:654] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 0f98711eaab94022940a3565f3a598c7: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:46.758628 20150 tablet_bootstrap.cc:654] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:46.759387 20148 log.cc:824] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 0f98711eaab94022940a3565f3a598c7: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:46.759819 20150 log.cc:824] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:46.760164 20149 tablet_bootstrap.cc:492] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 7973b491457a4e81a3bf9a8edcef52d6: Bootstrap starting.
I20260926 20:24:46.761356 20148 tablet_bootstrap.cc:492] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 0f98711eaab94022940a3565f3a598c7: No bootstrap required, opened a new log
I20260926 20:24:46.761570 20148 ts_tablet_manager.cc:1397] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 0f98711eaab94022940a3565f3a598c7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:24:46.761758 20150 tablet_bootstrap.cc:492] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87: No bootstrap required, opened a new log
I20260926 20:24:46.761940 20150 ts_tablet_manager.cc:1397] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:24:46.762245 20149 tablet_bootstrap.cc:654] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 7973b491457a4e81a3bf9a8edcef52d6: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:46.763036 20149 log.cc:824] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 7973b491457a4e81a3bf9a8edcef52d6: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:46.765189 20149 tablet_bootstrap.cc:492] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 7973b491457a4e81a3bf9a8edcef52d6: No bootstrap required, opened a new log
I20260926 20:24:46.765115 20148 raft_consensus.cc:348] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 0f98711eaab94022940a3565f3a598c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf6379a4394afe981f332f62c56b87" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 34907 } } peers { permanent_uuid: "7973b491457a4e81a3bf9a8edcef52d6" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 32817 } } peers { permanent_uuid: "0f98711eaab94022940a3565f3a598c7" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 44455 } }
I20260926 20:24:46.765430 20148 raft_consensus.cc:374] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 0f98711eaab94022940a3565f3a598c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:46.765496 20148 raft_consensus.cc:729] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 0f98711eaab94022940a3565f3a598c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f98711eaab94022940a3565f3a598c7, State: Initialized, Role: FOLLOWER
I20260926 20:24:46.765592 20149 ts_tablet_manager.cc:1397] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 7973b491457a4e81a3bf9a8edcef52d6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:24:46.766000 20148 consensus_queue.cc:260] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 0f98711eaab94022940a3565f3a598c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf6379a4394afe981f332f62c56b87" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 34907 } } peers { permanent_uuid: "7973b491457a4e81a3bf9a8edcef52d6" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 32817 } } peers { permanent_uuid: "0f98711eaab94022940a3565f3a598c7" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 44455 } }
I20260926 20:24:46.765903 20150 raft_consensus.cc:348] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf6379a4394afe981f332f62c56b87" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 34907 } } peers { permanent_uuid: "7973b491457a4e81a3bf9a8edcef52d6" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 32817 } } peers { permanent_uuid: "0f98711eaab94022940a3565f3a598c7" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 44455 } }
I20260926 20:24:46.766161 20150 raft_consensus.cc:374] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:46.766217 20150 raft_consensus.cc:729] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dbf6379a4394afe981f332f62c56b87, State: Initialized, Role: FOLLOWER
I20260926 20:24:46.766654 20150 consensus_queue.cc:260] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf6379a4394afe981f332f62c56b87" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 34907 } } peers { permanent_uuid: "7973b491457a4e81a3bf9a8edcef52d6" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 32817 } } peers { permanent_uuid: "0f98711eaab94022940a3565f3a598c7" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 44455 } }
I20260926 20:24:46.766983 20148 ts_tablet_manager.cc:1428] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 0f98711eaab94022940a3565f3a598c7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 20:24:46.767338 20131 heartbeater.cc:503] Master 127.0.116.254:44185 was elected leader, sending a full tablet report...
I20260926 20:24:46.767338 19997 heartbeater.cc:503] Master 127.0.116.254:44185 was elected leader, sending a full tablet report...
I20260926 20:24:46.767753 20150 ts_tablet_manager.cc:1428] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 20:24:46.770241 20149 raft_consensus.cc:348] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 7973b491457a4e81a3bf9a8edcef52d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf6379a4394afe981f332f62c56b87" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 34907 } } peers { permanent_uuid: "7973b491457a4e81a3bf9a8edcef52d6" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 32817 } } peers { permanent_uuid: "0f98711eaab94022940a3565f3a598c7" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 44455 } }
I20260926 20:24:46.770530 20149 raft_consensus.cc:374] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 7973b491457a4e81a3bf9a8edcef52d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:46.770599 20149 raft_consensus.cc:729] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 7973b491457a4e81a3bf9a8edcef52d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7973b491457a4e81a3bf9a8edcef52d6, State: Initialized, Role: FOLLOWER
I20260926 20:24:46.771062 20149 consensus_queue.cc:260] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 7973b491457a4e81a3bf9a8edcef52d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf6379a4394afe981f332f62c56b87" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 34907 } } peers { permanent_uuid: "7973b491457a4e81a3bf9a8edcef52d6" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 32817 } } peers { permanent_uuid: "0f98711eaab94022940a3565f3a598c7" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 44455 } }
I20260926 20:24:46.771768 19863 heartbeater.cc:503] Master 127.0.116.254:44185 was elected leader, sending a full tablet report...
I20260926 20:24:46.772214 20149 ts_tablet_manager.cc:1428] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 7973b491457a4e81a3bf9a8edcef52d6: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 20:24:46.895367 20155 raft_consensus.cc:482] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:24:46.895661 20155 raft_consensus.cc:504] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf6379a4394afe981f332f62c56b87" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 34907 } } peers { permanent_uuid: "7973b491457a4e81a3bf9a8edcef52d6" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 32817 } } peers { permanent_uuid: "0f98711eaab94022940a3565f3a598c7" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 44455 } }
I20260926 20:24:46.896875 20155 leader_election.cc:288] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7973b491457a4e81a3bf9a8edcef52d6 (127.0.116.193:32817), 0f98711eaab94022940a3565f3a598c7 (127.0.116.194:44455)
I20260926 20:24:46.900707 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:46.897222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:32817 (local address 127.0.116.195:45753)
0926 20:24:46.897359 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:46.897376 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:46.897469 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:46.897767 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:46.897770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:46.897788 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:46.897915 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:46.897920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.898724 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.898729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:46.899620 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:46.899630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.900267 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.900270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:46.900376 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:46.900398 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:24:46.900488 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 20:24:46.900567 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 20:24:46.900707 20145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:46.897355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:45753 (local address 127.0.116.193:32817)
0926 20:24:46.897512 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:46.897518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:46.897536 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:24:46.897596 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:46.897599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:46.897649 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:46.897740 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:46.898075 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.898603 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.899747 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.900149 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.900276 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:46.900335 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:46.900517 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 20:24:46.900595 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:24:46.901162 20147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:46.897618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:45127 (local address 127.0.116.194:44455)
0926 20:24:46.897764 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:46.897769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:46.898046 (+ 277us) server_negotiation.cc:407] Connection header received
0926 20:24:46.898182 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:46.898185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:46.898237 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:46.898319 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:46.898571 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.899063 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.900131 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.900544 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.900670 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:46.900717 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:46.900940 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 20:24:46.901051 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 20:24:46.901477 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:46.897515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:44455 (local address 127.0.116.195:45127)
0926 20:24:46.897953 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:46.897963 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:46.898084 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:46.898326 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:46.898329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:46.898344 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:46.898465 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:46.898471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.899169 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.899174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:46.900010 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:46.900019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.900660 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.900663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:46.900763 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:46.900806 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 20:24:46.901312 (+ 506us) client_negotiation.cc:240] Negotiation successful
0926 20:24:46.901380 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":158,"threads_started":1}
I20260926 20:24:46.901515 19817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61280d4839ff4f9bb9a3f92ad70fbc0c" candidate_uuid: "9dbf6379a4394afe981f332f62c56b87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7973b491457a4e81a3bf9a8edcef52d6" is_pre_election: true
I20260926 20:24:46.901881 19817 raft_consensus.cc:2454] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 7973b491457a4e81a3bf9a8edcef52d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dbf6379a4394afe981f332f62c56b87 in term 0.
I20260926 20:24:46.902213 19951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61280d4839ff4f9bb9a3f92ad70fbc0c" candidate_uuid: "9dbf6379a4394afe981f332f62c56b87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f98711eaab94022940a3565f3a598c7" is_pre_election: true
I20260926 20:24:46.902392 20020 leader_election.cc:302] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7973b491457a4e81a3bf9a8edcef52d6, 9dbf6379a4394afe981f332f62c56b87; no voters:
I20260926 20:24:46.902526 19951 raft_consensus.cc:2454] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 0f98711eaab94022940a3565f3a598c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dbf6379a4394afe981f332f62c56b87 in term 0.
I20260926 20:24:46.902642 20155 raft_consensus.cc:2781] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:24:46.902719 20155 raft_consensus.cc:482] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:24:46.902783 20155 raft_consensus.cc:3037] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:46.903971 20155 raft_consensus.cc:504] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf6379a4394afe981f332f62c56b87" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 34907 } } peers { permanent_uuid: "7973b491457a4e81a3bf9a8edcef52d6" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 32817 } } peers { permanent_uuid: "0f98711eaab94022940a3565f3a598c7" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 44455 } }
I20260926 20:24:46.904304 20155 leader_election.cc:288] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87 [CANDIDATE]: Term 1 election: Requested vote from peers 7973b491457a4e81a3bf9a8edcef52d6 (127.0.116.193:32817), 0f98711eaab94022940a3565f3a598c7 (127.0.116.194:44455)
I20260926 20:24:46.904702 19951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61280d4839ff4f9bb9a3f92ad70fbc0c" candidate_uuid: "9dbf6379a4394afe981f332f62c56b87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f98711eaab94022940a3565f3a598c7"
I20260926 20:24:46.904732 19817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61280d4839ff4f9bb9a3f92ad70fbc0c" candidate_uuid: "9dbf6379a4394afe981f332f62c56b87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7973b491457a4e81a3bf9a8edcef52d6"
I20260926 20:24:46.904809 19951 raft_consensus.cc:3037] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 0f98711eaab94022940a3565f3a598c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:46.904886 19817 raft_consensus.cc:3037] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 7973b491457a4e81a3bf9a8edcef52d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:46.905908 19951 raft_consensus.cc:2454] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 0f98711eaab94022940a3565f3a598c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dbf6379a4394afe981f332f62c56b87 in term 1.
I20260926 20:24:46.906023 19817 raft_consensus.cc:2454] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 7973b491457a4e81a3bf9a8edcef52d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dbf6379a4394afe981f332f62c56b87 in term 1.
I20260926 20:24:46.906247 20020 leader_election.cc:302] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f98711eaab94022940a3565f3a598c7, 9dbf6379a4394afe981f332f62c56b87; no voters:
I20260926 20:24:46.906415 20155 raft_consensus.cc:2781] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:24:46.906646 20155 raft_consensus.cc:686] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87 [term 1 LEADER]: Becoming Leader. State: Replica: 9dbf6379a4394afe981f332f62c56b87, State: Running, Role: LEADER
I20260926 20:24:46.907015 20155 consensus_queue.cc:237] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf6379a4394afe981f332f62c56b87" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 34907 } } peers { permanent_uuid: "7973b491457a4e81a3bf9a8edcef52d6" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 32817 } } peers { permanent_uuid: "0f98711eaab94022940a3565f3a598c7" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 44455 } }
I20260926 20:24:46.910506 19671 catalog_manager.cc:5495] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dbf6379a4394afe981f332f62c56b87 (127.0.116.195). New cstate: current_term: 1 leader_uuid: "9dbf6379a4394afe981f332f62c56b87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbf6379a4394afe981f332f62c56b87" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 34907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7973b491457a4e81a3bf9a8edcef52d6" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 32817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f98711eaab94022940a3565f3a598c7" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 44455 } health_report { overall_health: UNKNOWN } } }
W20260926 20:24:46.933377 20132 tablet.cc:2369] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:24:46.940831 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:46.937178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36250 (local address 127.0.116.195:34907)
0926 20:24:46.937322 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:46.937327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:46.937342 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:24:46.937458 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:46.937461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:46.937525 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:46.937629 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:24:46.937841 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.938530 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.939407 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:46.939669 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:46.939754 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:46.940072 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:24:46.940508 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:24:46.940611 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:46.940665 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 20:24:46.940718 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:24:46.946884 19951 raft_consensus.cc:1261] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 0f98711eaab94022940a3565f3a598c7 [term 1 FOLLOWER]: Refusing update from remote peer 9dbf6379a4394afe981f332f62c56b87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:24:46.946895 19817 raft_consensus.cc:1261] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 7973b491457a4e81a3bf9a8edcef52d6 [term 1 FOLLOWER]: Refusing update from remote peer 9dbf6379a4394afe981f332f62c56b87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:24:46.947703 20155 consensus_queue.cc:1035] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7973b491457a4e81a3bf9a8edcef52d6" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 32817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:24:46.947970 20158 consensus_queue.cc:1035] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f98711eaab94022940a3565f3a598c7" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 44455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:24:46.956972 20163 mvcc.cc:205] Tried to move back new op lower bound from 7333700759326830592 to 7333700759174434816. Current Snapshot: MvccSnapshot[applied={T|T < 7333700759326830592}]
I20260926 20:24:46.960062 20162 mvcc.cc:205] Tried to move back new op lower bound from 7333700759326830592 to 7333700759174434816. Current Snapshot: MvccSnapshot[applied={T|T < 7333700759326830592 or (T in {7333700759326830592})}]
I20260926 20:24:46.964516 19671 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:24:46.964702 19671 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:24:46.967172 19671 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 656f594ed94a460f88dcb2c8470f492f: Sending DeleteTablet for 3 replicas of tablet 61280d4839ff4f9bb9a3f92ad70fbc0c
I20260926 20:24:46.967964 20065 tablet_service.cc:1514] Processing DeleteTablet for tablet 61280d4839ff4f9bb9a3f92ad70fbc0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:24:46 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:36238
I20260926 20:24:46.968143 19931 tablet_service.cc:1514] Processing DeleteTablet for tablet 61280d4839ff4f9bb9a3f92ad70fbc0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:24:46 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:34226
I20260926 20:24:46.968206 19797 tablet_service.cc:1514] Processing DeleteTablet for tablet 61280d4839ff4f9bb9a3f92ad70fbc0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:24:46 UTC) from {username='kudu', principal='kudu/127.0.116.254@KRBTEST.COM'} at 127.0.0.1:33230
I20260926 20:24:46.968508 20170 tablet_replica.cc:330] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87: stopping tablet replica
I20260926 20:24:46.968793 20171 tablet_replica.cc:330] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 0f98711eaab94022940a3565f3a598c7: stopping tablet replica
I20260926 20:24:46.968825 20170 raft_consensus.cc:2229] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:24:46.969137 20172 tablet_replica.cc:330] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 7973b491457a4e81a3bf9a8edcef52d6: stopping tablet replica
I20260926 20:24:46.969208 20170 raft_consensus.cc:2258] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:24:46.969434 20172 raft_consensus.cc:2229] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 7973b491457a4e81a3bf9a8edcef52d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:24:46.969471 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 19734
I20260926 20:24:46.971061 20170 ts_tablet_manager.cc:1905] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:24:46.972664 20171 raft_consensus.cc:2229] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 0f98711eaab94022940a3565f3a598c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:24:46.973088 20171 raft_consensus.cc:2258] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 0f98711eaab94022940a3565f3a598c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:24:46.974229 20170 ts_tablet_manager.cc:1918] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:24:46.974334 20170 log.cc:1196] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/61280d4839ff4f9bb9a3f92ad70fbc0c
I20260926 20:24:46.974677 20170 ts_tablet_manager.cc:1939] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 9dbf6379a4394afe981f332f62c56b87: Deleting consensus metadata
I20260926 20:24:46.975000 20171 ts_tablet_manager.cc:1905] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 0f98711eaab94022940a3565f3a598c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:24:46.975973 19658 catalog_manager.cc:4841] TS 9dbf6379a4394afe981f332f62c56b87 (127.0.116.195:34907): tablet 61280d4839ff4f9bb9a3f92ad70fbc0c (table test-table [id=6b5c054a75e04b5fa4542d989aa0fd6c]) successfully deleted
I20260926 20:24:46.977589 20171 ts_tablet_manager.cc:1918] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 0f98711eaab94022940a3565f3a598c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:24:46.977706 20171 log.cc:1196] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 0f98711eaab94022940a3565f3a598c7: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/61280d4839ff4f9bb9a3f92ad70fbc0c
I20260926 20:24:46.978000 20171 ts_tablet_manager.cc:1939] T 61280d4839ff4f9bb9a3f92ad70fbc0c P 0f98711eaab94022940a3565f3a598c7: Deleting consensus metadata
I20260926 20:24:46.978866 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 19868
W20260926 20:24:46.978840 19658 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.116.193:32817 (error 108)
I20260926 20:24:46.983203 20146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:46.982923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:32817 (local address 127.0.0.1:33234)
0926 20:24:46.983049 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:46.983127 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.193:32817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 20:24:46.983477 19658 catalog_manager.cc:4584] TS 7973b491457a4e81a3bf9a8edcef52d6 (127.0.116.193:32817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 61280d4839ff4f9bb9a3f92ad70fbc0c: Network error: Client connection negotiation failed: client connection to 127.0.116.193:32817: connect: Connection refused (error 111)
I20260926 20:24:46.986266 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 20002
I20260926 20:24:46.988236 20146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:46.987976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:44455 (local address 127.0.0.1:34242)
0926 20:24:46.988100 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:46.988162 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.194:44455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 20:24:46.991032 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 19640
2026-09-26T20:24:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2459 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T20:24:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:24:47Z Disabled control of system clock
I20260926 20:24:47.021203 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:46525
--webserver_interface=127.0.116.254
--webserver_port=0
--builtin_ntp_servers=127.0.116.212:45193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:46525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:24:47.133946 20177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:47.134246 20177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:47.134284 20177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:47.138602 20177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:24:47.138676 20177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:47.138691 20177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:47.138710 20177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:24:47.138726 20177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:24:47.143788 20177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:45193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:46525
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:46525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:47.145009 20177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:47.146481 20177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:47.155369 20185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:47.155362 20183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:47.155385 20182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:47.155565 20177 server_base.cc:1034] running on GCE node
I20260926 20:24:47.156519 20177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:47.157770 20177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:47.158978 20177 hybrid_clock.cc:648] HybridClock initialized: now 1790454287158940 us; error 52 us; skew 500 ppm
I20260926 20:24:47.161091 20177 webserver.cc:460] Webserver started at http://127.0.116.254:39231/ using document root <none> and password file <none>
I20260926 20:24:47.161708 20177 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:47.161773 20177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:47.162009 20177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:47.164247 20177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "34c9f77634ae4579acf1bd596122d2df"
format_stamp: "Formatted at 2026-09-26 20:24:47 on dist-test-slave-qsm9"
server_key: "ce47a403edc062550dbeadad9a42d262"
server_key_iv: "2212ac7a1dcf0d827d6241d2fc3c102a"
server_key_version: "encryptionkey@0"
I20260926 20:24:47.164832 20177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "34c9f77634ae4579acf1bd596122d2df"
format_stamp: "Formatted at 2026-09-26 20:24:47 on dist-test-slave-qsm9"
server_key: "ce47a403edc062550dbeadad9a42d262"
server_key_iv: "2212ac7a1dcf0d827d6241d2fc3c102a"
server_key_version: "encryptionkey@0"
I20260926 20:24:47.168766 20177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:24:47.171202 20191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:47.172551 20177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:24:47.172693 20177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "34c9f77634ae4579acf1bd596122d2df"
format_stamp: "Formatted at 2026-09-26 20:24:47 on dist-test-slave-qsm9"
server_key: "ce47a403edc062550dbeadad9a42d262"
server_key_iv: "2212ac7a1dcf0d827d6241d2fc3c102a"
server_key_version: "encryptionkey@0"
I20260926 20:24:47.172807 20177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:47.209173 20177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:47.210080 20177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:47.210294 20177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:47.219130 20177 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:46525
I20260926 20:24:47.219175 20243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:46525 every 8 connection(s)
I20260926 20:24:47.220218 20177 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:24:47.224838 20244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:47.227739 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 20177
I20260926 20:24:47.227870 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:24:47.228183 467 external_mini_cluster.cc:1428] Setting key e46d8e29c7ea487f27948787b068f848
I20260926 20:24:47.232827 20244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df: Bootstrap starting.
I20260926 20:24:47.235543 20244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:47.236295 20244 log.cc:824] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:47.238296 20244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df: No bootstrap required, opened a new log
I20260926 20:24:47.241943 20244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c9f77634ae4579acf1bd596122d2df" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46525 } }
I20260926 20:24:47.242229 20244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:47.242302 20244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34c9f77634ae4579acf1bd596122d2df, State: Initialized, Role: FOLLOWER
I20260926 20:24:47.242823 20244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c9f77634ae4579acf1bd596122d2df" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46525 } }
I20260926 20:24:47.242990 20244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:24:47.243058 20244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:24:47.243156 20244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:47.244575 20244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c9f77634ae4579acf1bd596122d2df" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46525 } }
I20260926 20:24:47.244971 20244 leader_election.cc:302] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34c9f77634ae4579acf1bd596122d2df; no voters:
I20260926 20:24:47.245344 20244 leader_election.cc:288] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:24:47.245399 20247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:47.229757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.0.116.254:46525)
0926 20:24:47.230397 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:47.230413 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:47.230455 (+ 42us) server_negotiation.cc:407] Connection header received
0926 20:24:47.231110 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:47.231127 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:47.231587 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:47.231876 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:47.232252 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.233320 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.234531 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.234985 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.235097 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:47.242845 (+ 7748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:47.242852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:47.242871 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:47.243464 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:47.243602 (+ 138us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:47.243636 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:47.243712 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:47.244003 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 20:24:47.244332 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":138,"threads_started":1}
I20260926 20:24:47.245481 20249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:24:47.245751 20249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df [term 1 LEADER]: Becoming Leader. State: Replica: 34c9f77634ae4579acf1bd596122d2df, State: Running, Role: LEADER
I20260926 20:24:47.246140 20249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c9f77634ae4579acf1bd596122d2df" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46525 } }
I20260926 20:24:47.246671 20244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:24:47.248479 20250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34c9f77634ae4579acf1bd596122d2df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c9f77634ae4579acf1bd596122d2df" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46525 } } }
I20260926 20:24:47.248631 20250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:47.248996 20263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:24:47.248963 20251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34c9f77634ae4579acf1bd596122d2df. Latest consensus state: current_term: 1 leader_uuid: "34c9f77634ae4579acf1bd596122d2df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c9f77634ae4579acf1bd596122d2df" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 46525 } } }
I20260926 20:24:47.249137 20251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:47.252106 20263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:24:47.258567 20263 catalog_manager.cc:1347] Generated new cluster ID: d67cbaf1722b403f89e6cd18987bdb8e
I20260926 20:24:47.258658 20263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:24:47.287978 20263 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:24:47.289448 20263 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:24:47.298262 20263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df: Generated new TSK 0
I20260926 20:24:47.299054 20263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:24:47.301976 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.193:0
--local_ip_for_outbound_sockets=127.0.116.193
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46525
--builtin_ntp_servers=127.0.116.212:45193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:24:47.433681 20268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:47.433969 20268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:47.434007 20268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:47.438171 20268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:47.438241 20268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:47.438297 20268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.193
I20260926 20:24:47.443440 20268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:45193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.116.193
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.193
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:47.444629 20268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:47.446089 20268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:24:47.456128 20268 server_base.cc:1034] running on GCE node
W20260926 20:24:47.456056 20276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:47.456085 20273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:47.456051 20274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:47.456810 20268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:47.457540 20268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:47.458748 20268 hybrid_clock.cc:648] HybridClock initialized: now 1790454287458724 us; error 38 us; skew 500 ppm
I20260926 20:24:47.460829 20268 webserver.cc:460] Webserver started at http://127.0.116.193:35933/ using document root <none> and password file <none>
I20260926 20:24:47.461477 20268 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:47.461575 20268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:47.461813 20268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:47.464123 20268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-0/data/instance:
uuid: "8d43a42d7b0141d98900a55c744c2b0f"
format_stamp: "Formatted at 2026-09-26 20:24:47 on dist-test-slave-qsm9"
server_key: "208933ef021f042ab83c0012b905e590"
server_key_iv: "eb90a4d054cd102755568b7866f1b0f0"
server_key_version: "encryptionkey@0"
I20260926 20:24:47.464735 20268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-0/wal/instance:
uuid: "8d43a42d7b0141d98900a55c744c2b0f"
format_stamp: "Formatted at 2026-09-26 20:24:47 on dist-test-slave-qsm9"
server_key: "208933ef021f042ab83c0012b905e590"
server_key_iv: "eb90a4d054cd102755568b7866f1b0f0"
server_key_version: "encryptionkey@0"
I20260926 20:24:47.468695 20268 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 20:24:47.471246 20282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:47.472632 20268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:24:47.472769 20268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-0/wal
uuid: "8d43a42d7b0141d98900a55c744c2b0f"
format_stamp: "Formatted at 2026-09-26 20:24:47 on dist-test-slave-qsm9"
server_key: "208933ef021f042ab83c0012b905e590"
server_key_iv: "eb90a4d054cd102755568b7866f1b0f0"
server_key_version: "encryptionkey@0"
I20260926 20:24:47.472888 20268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:47.491395 20268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:47.492237 20268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:47.492445 20268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:47.493062 20268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:47.494042 20268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:24:47.494119 20268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:47.494205 20268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:24:47.494251 20268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:47.503407 20268 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.193:33775
I20260926 20:24:47.503455 20395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.193:33775 every 8 connection(s)
I20260926 20:24:47.504464 20268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-0/data/info.pb
I20260926 20:24:47.511443 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 20268
I20260926 20:24:47.511636 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-0/wal/instance
I20260926 20:24:47.511956 467 external_mini_cluster.cc:1428] Setting key 0aa319c528352e0092162a38932fcfba
I20260926 20:24:47.514103 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.194:0
--local_ip_for_outbound_sockets=127.0.116.194
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46525
--builtin_ntp_servers=127.0.116.212:45193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:24:47.515090 20247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:47.506435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:57489 (local address 127.0.116.254:46525)
0926 20:24:47.506579 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:47.506583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:47.507724 (+ 1141us) server_negotiation.cc:407] Connection header received
0926 20:24:47.509570 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:47.509573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:47.509625 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:47.509707 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:47.510704 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.511334 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.512862 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.513169 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.513298 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:47.514558 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:47.514562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:47.514564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:47.514579 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:47.514606 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:47.514614 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:47.514688 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:47.514856 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 20:24:47.514942 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:24:47.516176 20398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:47.506825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46525 (local address 127.0.116.193:57489)
0926 20:24:47.507479 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:47.507606 (+ 127us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:47.509327 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:47.509930 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:47.509938 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:47.510033 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:47.510552 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:47.510566 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.511558 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.511563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:47.512729 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:47.512747 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.513331 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.513335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:47.513521 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:47.514234 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:24:47.514237 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:47.514314 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:24:47.514391 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:47.514703 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:47.514712 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:47.514718 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:24:47.514858 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 20:24:47.515286 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":101,"threads_started":1}
I20260926 20:24:47.517505 20396 heartbeater.cc:347] Connected to a master server at 127.0.116.254:46525
I20260926 20:24:47.517788 20396 heartbeater.cc:464] Registering TS with master...
I20260926 20:24:47.518522 20396 heartbeater.cc:511] Master 127.0.116.254:46525 requested a full tablet report, sending...
I20260926 20:24:47.520273 20208 ts_manager.cc:194] Registered new tserver with Master: 8d43a42d7b0141d98900a55c744c2b0f (127.0.116.193:33775)
I20260926 20:24:47.521492 20208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.116.193:57489
W20260926 20:24:47.630080 20399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:47.630419 20399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:47.630503 20399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:47.634656 20399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:47.634728 20399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:47.634826 20399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.194
I20260926 20:24:47.640286 20399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:45193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.116.194
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.194
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:47.641578 20399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:47.643081 20399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:47.653692 20407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:47.653704 20404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:47.653708 20405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:47.653918 20399 server_base.cc:1034] running on GCE node
I20260926 20:24:47.654456 20399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:47.655208 20399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:47.656436 20399 hybrid_clock.cc:648] HybridClock initialized: now 1790454287656390 us; error 57 us; skew 500 ppm
I20260926 20:24:47.658473 20399 webserver.cc:460] Webserver started at http://127.0.116.194:43359/ using document root <none> and password file <none>
I20260926 20:24:47.659119 20399 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:47.659220 20399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:47.659473 20399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:47.661864 20399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-1/data/instance:
uuid: "c5158881a11941c8aeda1b110bb4a606"
format_stamp: "Formatted at 2026-09-26 20:24:47 on dist-test-slave-qsm9"
server_key: "995aef3a82a2df44d5f694e21792f9ff"
server_key_iv: "aab5f3343d98290e84973d1c27730972"
server_key_version: "encryptionkey@0"
I20260926 20:24:47.662482 20399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-1/wal/instance:
uuid: "c5158881a11941c8aeda1b110bb4a606"
format_stamp: "Formatted at 2026-09-26 20:24:47 on dist-test-slave-qsm9"
server_key: "995aef3a82a2df44d5f694e21792f9ff"
server_key_iv: "aab5f3343d98290e84973d1c27730972"
server_key_version: "encryptionkey@0"
I20260926 20:24:47.666829 20399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 20:24:47.669492 20413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:47.670828 20399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:24:47.670987 20399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-1/wal
uuid: "c5158881a11941c8aeda1b110bb4a606"
format_stamp: "Formatted at 2026-09-26 20:24:47 on dist-test-slave-qsm9"
server_key: "995aef3a82a2df44d5f694e21792f9ff"
server_key_iv: "aab5f3343d98290e84973d1c27730972"
server_key_version: "encryptionkey@0"
I20260926 20:24:47.671103 20399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:47.696718 20399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:47.697474 20399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:47.697674 20399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:47.698283 20399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:47.699270 20399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:24:47.699338 20399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:47.699409 20399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:24:47.699460 20399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:47.708681 20399 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.194:44421
I20260926 20:24:47.708709 20526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.194:44421 every 8 connection(s)
I20260926 20:24:47.709664 20399 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-1/data/info.pb
I20260926 20:24:47.711505 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 20399
I20260926 20:24:47.711649 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-1/wal/instance
I20260926 20:24:47.711994 467 external_mini_cluster.cc:1428] Setting key b370c510a888f56effdcbec83db8d3d5
I20260926 20:24:47.714150 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.116.195:0
--local_ip_for_outbound_sockets=127.0.116.195
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46525
--builtin_ntp_servers=127.0.116.212:45193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:24:47.720130 20247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:47.711830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.194:42807 (local address 127.0.116.254:46525)
0926 20:24:47.711971 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:47.711975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:47.713411 (+ 1436us) server_negotiation.cc:407] Connection header received
0926 20:24:47.714933 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:47.714936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:47.714999 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:47.715105 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:47.716124 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.716666 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.717874 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.718105 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.718222 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:47.719567 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:47.719575 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:47.719577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:47.719592 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:47.719613 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:47.719622 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:47.719710 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:47.719893 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 20:24:47.719974 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:24:47.721107 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:47.712381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46525 (local address 127.0.116.194:42807)
0926 20:24:47.713239 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:47.713281 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:47.714714 (+ 1433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:47.715306 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:47.715314 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:47.715407 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:47.715978 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:47.715991 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.716807 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.716812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:47.717737 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:47.717748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.718274 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.718279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:47.718460 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:47.719241 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:24:47.719242 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:47.719289 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:24:47.719364 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:47.719714 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:47.719726 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:47.719733 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 20:24:47.719871 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 20:24:47.720252 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":165,"threads_started":1}
I20260926 20:24:47.722409 20527 heartbeater.cc:347] Connected to a master server at 127.0.116.254:46525
I20260926 20:24:47.722734 20527 heartbeater.cc:464] Registering TS with master...
I20260926 20:24:47.723555 20527 heartbeater.cc:511] Master 127.0.116.254:46525 requested a full tablet report, sending...
I20260926 20:24:47.724895 20208 ts_manager.cc:194] Registered new tserver with Master: c5158881a11941c8aeda1b110bb4a606 (127.0.116.194:44421)
I20260926 20:24:47.725929 20208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.116.194:42807
W20260926 20:24:47.830811 20530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:47.831084 20530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:47.831123 20530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:47.835211 20530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:47.835273 20530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:47.835330 20530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.116.195
I20260926 20:24:47.840126 20530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:45193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.116.195
--webserver_port=0
--tserver_master_addrs=127.0.116.254:46525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.116.195
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:47.841154 20530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:47.842453 20530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:47.852236 20536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:47.852222 20538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:47.852221 20535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:47.852571 20530 server_base.cc:1034] running on GCE node
I20260926 20:24:47.852995 20530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:47.853684 20530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:47.854878 20530 hybrid_clock.cc:648] HybridClock initialized: now 1790454287854839 us; error 50 us; skew 500 ppm
I20260926 20:24:47.856848 20530 webserver.cc:460] Webserver started at http://127.0.116.195:37747/ using document root <none> and password file <none>
I20260926 20:24:47.857436 20530 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:47.857527 20530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:47.857748 20530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:47.859951 20530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-2/data/instance:
uuid: "81b4acfb70a94b3aa237417a50e618f0"
format_stamp: "Formatted at 2026-09-26 20:24:47 on dist-test-slave-qsm9"
server_key: "7259d850155d46031ec1c9637d629987"
server_key_iv: "b69fee4ca4cb138b87115b1d069a3662"
server_key_version: "encryptionkey@0"
I20260926 20:24:47.860520 20530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-2/wal/instance:
uuid: "81b4acfb70a94b3aa237417a50e618f0"
format_stamp: "Formatted at 2026-09-26 20:24:47 on dist-test-slave-qsm9"
server_key: "7259d850155d46031ec1c9637d629987"
server_key_iv: "b69fee4ca4cb138b87115b1d069a3662"
server_key_version: "encryptionkey@0"
I20260926 20:24:47.864333 20530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:24:47.866889 20544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:47.868201 20530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:24:47.868345 20530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-2/wal
uuid: "81b4acfb70a94b3aa237417a50e618f0"
format_stamp: "Formatted at 2026-09-26 20:24:47 on dist-test-slave-qsm9"
server_key: "7259d850155d46031ec1c9637d629987"
server_key_iv: "b69fee4ca4cb138b87115b1d069a3662"
server_key_version: "encryptionkey@0"
I20260926 20:24:47.868459 20530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:47.899169 20530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:47.900028 20530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:47.900244 20530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:47.900892 20530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:24:47.901949 20530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:24:47.901998 20530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:47.902060 20530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:24:47.902093 20530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:47.911751 20530 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.195:45093
I20260926 20:24:47.911791 20657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.195:45093 every 8 connection(s)
I20260926 20:24:47.912782 20530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-2/data/info.pb
I20260926 20:24:47.921566 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 20530
I20260926 20:24:47.921712 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-2/wal/instance
I20260926 20:24:47.922010 467 external_mini_cluster.cc:1428] Setting key 5873f27a3f776c2934ebe3495748b3ad
I20260926 20:24:47.922824 20247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:47.914694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.195:46921 (local address 127.0.116.254:46525)
0926 20:24:47.914855 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:47.914860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:47.915991 (+ 1131us) server_negotiation.cc:407] Connection header received
0926 20:24:47.917499 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:47.917503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:47.917575 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:47.917685 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:47.918660 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.919230 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.920505 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.920752 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.920887 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:47.922246 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:47.922250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:47.922252 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:47.922268 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:47.922308 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:47.922322 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:47.922408 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:47.922605 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 20:24:47.922677 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:24:47.923878 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:47.915065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:46525 (local address 127.0.116.195:46921)
0926 20:24:47.915803 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:47.915854 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:47.917239 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:47.917879 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:47.917887 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:47.917982 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:24:47.918503 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:47.918516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.919379 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.919384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:47.920366 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:47.920376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.920934 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.920938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:47.921113 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:47.921893 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:24:47.921894 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:24:47.921939 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:24:47.922011 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:24:47.922425 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:24:47.922433 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:24:47.922439 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:24:47.922605 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 20:24:47.922979 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":108,"threads_started":1}
I20260926 20:24:47.925012 20658 heartbeater.cc:347] Connected to a master server at 127.0.116.254:46525
I20260926 20:24:47.925287 20658 heartbeater.cc:464] Registering TS with master...
I20260926 20:24:47.925846 20658 heartbeater.cc:511] Master 127.0.116.254:46525 requested a full tablet report, sending...
I20260926 20:24:47.926909 20208 ts_manager.cc:194] Registered new tserver with Master: 81b4acfb70a94b3aa237417a50e618f0 (127.0.116.195:45093)
I20260926 20:24:47.927475 20208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.116.195:46921
I20260926 20:24:47.936396 467 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:24:47.942271 20247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:47.938459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53954 (local address 127.0.116.254:46525)
0926 20:24:47.938584 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:47.938587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:47.938776 (+ 189us) server_negotiation.cc:407] Connection header received
0926 20:24:47.938945 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:47.938948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:47.938994 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:47.939070 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:47.939314 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.939950 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.940786 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.941029 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.941112 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:47.941925 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:47.941928 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:47.941930 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:47.941942 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:47.941959 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:47.941966 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:47.942024 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:47.942114 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 20:24:47.942162 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:24:47.945046 20208 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:24:47.948310 20208 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:24:47.963416 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:47.956945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43204 (local address 127.0.116.194:44421)
0926 20:24:47.957719 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:47.957732 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:47.958336 (+ 604us) server_negotiation.cc:407] Connection header received
0926 20:24:47.958422 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:47.958437 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:47.958672 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:47.958824 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:47.959112 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.960147 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.961544 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.962375 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.962505 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:47.962733 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:47.963117 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 20:24:47.963225 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":101,"threads_started":1}
I20260926 20:24:47.963996 20672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:47.957309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39148 (local address 127.0.116.195:45093)
0926 20:24:47.957813 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:47.957826 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:47.958139 (+ 313us) server_negotiation.cc:407] Connection header received
0926 20:24:47.958415 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:47.958421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:47.958670 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:47.958797 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:47.959315 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.960501 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.962233 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.963017 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.963110 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:47.963285 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:47.963671 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 20:24:47.963836 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":90,"threads_started":1}
I20260926 20:24:47.964118 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:47.957887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:33775 (local address 127.0.0.1:44138)
0926 20:24:47.958442 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:47.958455 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:47.958541 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:47.958985 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:47.958988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:47.959003 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:47.959110 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:47.959116 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.960625 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.960628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:47.962089 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:47.962100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.963636 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.963639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:47.963723 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:47.963744 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 20:24:47.963819 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:24:47.963950 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":165,"thread_start_us":96,"threads_started":1}
I20260926 20:24:47.964157 20671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:47.956849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:44421 (local address 127.0.0.1:43204)
0926 20:24:47.958003 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:47.958053 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:47.958193 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:47.958816 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:47.958822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:47.958845 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:47.958991 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:47.958998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.960272 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.960275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:47.961420 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:47.961430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.962800 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.962804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:47.962925 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:47.962965 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 20:24:47.963084 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 20:24:47.963989 (+ 905us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"mutex_wait_us":91,"thread_start_us":126,"threads_started":1}
I20260926 20:24:47.964371 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:47.956856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:45093 (local address 127.0.0.1:39148)
0926 20:24:47.958003 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:47.958054 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:47.958240 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:47.958995 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:47.958998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:47.959026 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:47.959148 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:47.959158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.960625 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.960628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:47.962108 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:47.962114 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.963383 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.963386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:47.963483 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:47.963531 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 20:24:47.964178 (+ 647us) client_negotiation.cc:240] Negotiation successful
0926 20:24:47.964258 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"mutex_wait_us":277,"spinlock_wait_cycles":3328,"thread_start_us":80,"threads_started":1}
I20260926 20:24:47.964562 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:47.958048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44138 (local address 127.0.116.193:33775)
0926 20:24:47.958412 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:47.958423 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:47.958543 (+ 120us) server_negotiation.cc:407] Connection header received
0926 20:24:47.958647 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:47.958652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:47.958805 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:47.958947 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:47.959227 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.960484 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.962231 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:47.963017 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:47.963360 (+ 343us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:47.963574 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:47.964274 (+ 700us) server_negotiation.cc:299] Negotiation successful
0926 20:24:47.964369 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":128,"threads_started":1}
I20260926 20:24:47.967548 20592 tablet_service.cc:1467] Processing CreateTablet for tablet 2338a83ad859401db65012695d7eb2b6 (DEFAULT_TABLE table=test-table [id=01da925a836a4ca8b7a91be0f128c97b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:24:47.967564 20329 tablet_service.cc:1467] Processing CreateTablet for tablet 2338a83ad859401db65012695d7eb2b6 (DEFAULT_TABLE table=test-table [id=01da925a836a4ca8b7a91be0f128c97b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:24:47.967555 20461 tablet_service.cc:1467] Processing CreateTablet for tablet 2338a83ad859401db65012695d7eb2b6 (DEFAULT_TABLE table=test-table [id=01da925a836a4ca8b7a91be0f128c97b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:24:47.968670 20592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2338a83ad859401db65012695d7eb2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:47.968727 20329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2338a83ad859401db65012695d7eb2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:47.968727 20461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2338a83ad859401db65012695d7eb2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:47.974949 20675 tablet_bootstrap.cc:492] T 2338a83ad859401db65012695d7eb2b6 P 81b4acfb70a94b3aa237417a50e618f0: Bootstrap starting.
I20260926 20:24:47.975848 20676 tablet_bootstrap.cc:492] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f: Bootstrap starting.
I20260926 20:24:47.976907 20677 tablet_bootstrap.cc:492] T 2338a83ad859401db65012695d7eb2b6 P c5158881a11941c8aeda1b110bb4a606: Bootstrap starting.
I20260926 20:24:47.978288 20676 tablet_bootstrap.cc:654] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:47.978634 20675 tablet_bootstrap.cc:654] T 2338a83ad859401db65012695d7eb2b6 P 81b4acfb70a94b3aa237417a50e618f0: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:47.979044 20677 tablet_bootstrap.cc:654] T 2338a83ad859401db65012695d7eb2b6 P c5158881a11941c8aeda1b110bb4a606: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:47.979264 20676 log.cc:824] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:47.979341 20675 log.cc:824] T 2338a83ad859401db65012695d7eb2b6 P 81b4acfb70a94b3aa237417a50e618f0: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:47.979981 20677 log.cc:824] T 2338a83ad859401db65012695d7eb2b6 P c5158881a11941c8aeda1b110bb4a606: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:47.981393 20675 tablet_bootstrap.cc:492] T 2338a83ad859401db65012695d7eb2b6 P 81b4acfb70a94b3aa237417a50e618f0: No bootstrap required, opened a new log
I20260926 20:24:47.981403 20676 tablet_bootstrap.cc:492] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f: No bootstrap required, opened a new log
I20260926 20:24:47.981592 20675 ts_tablet_manager.cc:1397] T 2338a83ad859401db65012695d7eb2b6 P 81b4acfb70a94b3aa237417a50e618f0: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 20:24:47.981592 20676 ts_tablet_manager.cc:1397] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:24:47.981838 20677 tablet_bootstrap.cc:492] T 2338a83ad859401db65012695d7eb2b6 P c5158881a11941c8aeda1b110bb4a606: No bootstrap required, opened a new log
I20260926 20:24:47.982036 20677 ts_tablet_manager.cc:1397] T 2338a83ad859401db65012695d7eb2b6 P c5158881a11941c8aeda1b110bb4a606: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:24:47.985682 20677 raft_consensus.cc:348] T 2338a83ad859401db65012695d7eb2b6 P c5158881a11941c8aeda1b110bb4a606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5158881a11941c8aeda1b110bb4a606" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 44421 } } peers { permanent_uuid: "81b4acfb70a94b3aa237417a50e618f0" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 45093 } } peers { permanent_uuid: "8d43a42d7b0141d98900a55c744c2b0f" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33775 } }
I20260926 20:24:47.985975 20677 raft_consensus.cc:374] T 2338a83ad859401db65012695d7eb2b6 P c5158881a11941c8aeda1b110bb4a606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:47.986056 20677 raft_consensus.cc:729] T 2338a83ad859401db65012695d7eb2b6 P c5158881a11941c8aeda1b110bb4a606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5158881a11941c8aeda1b110bb4a606, State: Initialized, Role: FOLLOWER
I20260926 20:24:47.986011 20676 raft_consensus.cc:348] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5158881a11941c8aeda1b110bb4a606" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 44421 } } peers { permanent_uuid: "81b4acfb70a94b3aa237417a50e618f0" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 45093 } } peers { permanent_uuid: "8d43a42d7b0141d98900a55c744c2b0f" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33775 } }
I20260926 20:24:47.986277 20676 raft_consensus.cc:374] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:47.986389 20676 raft_consensus.cc:729] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d43a42d7b0141d98900a55c744c2b0f, State: Initialized, Role: FOLLOWER
I20260926 20:24:47.986331 20675 raft_consensus.cc:348] T 2338a83ad859401db65012695d7eb2b6 P 81b4acfb70a94b3aa237417a50e618f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5158881a11941c8aeda1b110bb4a606" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 44421 } } peers { permanent_uuid: "81b4acfb70a94b3aa237417a50e618f0" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 45093 } } peers { permanent_uuid: "8d43a42d7b0141d98900a55c744c2b0f" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33775 } }
I20260926 20:24:47.986564 20675 raft_consensus.cc:374] T 2338a83ad859401db65012695d7eb2b6 P 81b4acfb70a94b3aa237417a50e618f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:47.986670 20675 raft_consensus.cc:729] T 2338a83ad859401db65012695d7eb2b6 P 81b4acfb70a94b3aa237417a50e618f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b4acfb70a94b3aa237417a50e618f0, State: Initialized, Role: FOLLOWER
I20260926 20:24:47.986644 20677 consensus_queue.cc:260] T 2338a83ad859401db65012695d7eb2b6 P c5158881a11941c8aeda1b110bb4a606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5158881a11941c8aeda1b110bb4a606" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 44421 } } peers { permanent_uuid: "81b4acfb70a94b3aa237417a50e618f0" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 45093 } } peers { permanent_uuid: "8d43a42d7b0141d98900a55c744c2b0f" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33775 } }
I20260926 20:24:47.986835 20676 consensus_queue.cc:260] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5158881a11941c8aeda1b110bb4a606" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 44421 } } peers { permanent_uuid: "81b4acfb70a94b3aa237417a50e618f0" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 45093 } } peers { permanent_uuid: "8d43a42d7b0141d98900a55c744c2b0f" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33775 } }
I20260926 20:24:47.987109 20675 consensus_queue.cc:260] T 2338a83ad859401db65012695d7eb2b6 P 81b4acfb70a94b3aa237417a50e618f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5158881a11941c8aeda1b110bb4a606" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 44421 } } peers { permanent_uuid: "81b4acfb70a94b3aa237417a50e618f0" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 45093 } } peers { permanent_uuid: "8d43a42d7b0141d98900a55c744c2b0f" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33775 } }
I20260926 20:24:47.987437 20527 heartbeater.cc:503] Master 127.0.116.254:46525 was elected leader, sending a full tablet report...
I20260926 20:24:47.987831 20677 ts_tablet_manager.cc:1428] T 2338a83ad859401db65012695d7eb2b6 P c5158881a11941c8aeda1b110bb4a606: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 20:24:47.987838 20676 ts_tablet_manager.cc:1428] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 20:24:47.988200 20396 heartbeater.cc:503] Master 127.0.116.254:46525 was elected leader, sending a full tablet report...
I20260926 20:24:47.989068 20658 heartbeater.cc:503] Master 127.0.116.254:46525 was elected leader, sending a full tablet report...
I20260926 20:24:47.990828 20675 ts_tablet_manager.cc:1428] T 2338a83ad859401db65012695d7eb2b6 P 81b4acfb70a94b3aa237417a50e618f0: Time spent starting tablet: real 0.009s user 0.002s sys 0.005s
W20260926 20:24:48.005724 20397 tablet.cc:2369] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:24:48.080968 20682 raft_consensus.cc:482] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:24:48.081168 20682 raft_consensus.cc:504] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5158881a11941c8aeda1b110bb4a606" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 44421 } } peers { permanent_uuid: "81b4acfb70a94b3aa237417a50e618f0" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 45093 } } peers { permanent_uuid: "8d43a42d7b0141d98900a55c744c2b0f" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33775 } }
I20260926 20:24:48.082289 20682 leader_election.cc:288] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5158881a11941c8aeda1b110bb4a606 (127.0.116.194:44421), 81b4acfb70a94b3aa237417a50e618f0 (127.0.116.195:45093)
I20260926 20:24:48.087976 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:48.082688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:55647 (local address 127.0.116.194:44421)
0926 20:24:48.082895 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:48.082901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:48.083103 (+ 202us) server_negotiation.cc:407] Connection header received
0926 20:24:48.083199 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:48.083202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:48.083262 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:48.083358 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:48.083882 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:48.084695 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:48.086537 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:48.087176 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:48.087329 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:48.087370 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:48.087740 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 20:24:48.087830 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:24:48.087978 20672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:48.082672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.116.193:41101 (local address 127.0.116.195:45093)
0926 20:24:48.082798 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:48.082805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:48.083058 (+ 253us) server_negotiation.cc:407] Connection header received
0926 20:24:48.083202 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:48.083205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:48.083261 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:48.083358 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:24:48.083882 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:48.084695 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:48.086537 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:48.087190 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:48.087311 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:48.087368 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:48.087740 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 20:24:48.087880 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:24:48.088048 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:48.082582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.195:45093 (local address 127.0.116.193:41101)
0926 20:24:48.082951 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:48.082966 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:48.083093 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:48.083381 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:48.083384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:48.083431 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:48.083702 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:48.083711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:48.084828 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:48.084832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:48.086381 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:48.086394 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:48.087315 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:48.087319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:48.087584 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:48.087621 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 20:24:48.087689 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 20:24:48.087804 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":304,"spinlock_wait_cycles":28288,"thread_start_us":99,"threads_started":1}
I20260926 20:24:48.088048 20684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:48.082594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:44421 (local address 127.0.116.193:55647)
0926 20:24:48.083010 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:48.083025 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:24:48.083098 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:24:48.083381 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:24:48.083384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:24:48.083427 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:24:48.083664 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:48.083673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:48.084828 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:48.084832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:48.086381 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:24:48.086394 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:24:48.087315 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:24:48.087319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:24:48.087555 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:48.087608 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 20:24:48.087689 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 20:24:48.087804 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":245,"spinlock_wait_cycles":16768,"thread_start_us":141,"threads_started":1}
I20260926 20:24:48.088776 20612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2338a83ad859401db65012695d7eb2b6" candidate_uuid: "8d43a42d7b0141d98900a55c744c2b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b4acfb70a94b3aa237417a50e618f0" is_pre_election: true
I20260926 20:24:48.089051 20481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2338a83ad859401db65012695d7eb2b6" candidate_uuid: "8d43a42d7b0141d98900a55c744c2b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5158881a11941c8aeda1b110bb4a606" is_pre_election: true
I20260926 20:24:48.089115 20612 raft_consensus.cc:2454] T 2338a83ad859401db65012695d7eb2b6 P 81b4acfb70a94b3aa237417a50e618f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d43a42d7b0141d98900a55c744c2b0f in term 0.
I20260926 20:24:48.089342 20481 raft_consensus.cc:2454] T 2338a83ad859401db65012695d7eb2b6 P c5158881a11941c8aeda1b110bb4a606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d43a42d7b0141d98900a55c744c2b0f in term 0.
I20260926 20:24:48.089619 20285 leader_election.cc:302] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81b4acfb70a94b3aa237417a50e618f0, 8d43a42d7b0141d98900a55c744c2b0f; no voters:
I20260926 20:24:48.089916 20682 raft_consensus.cc:2781] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:24:48.089998 20682 raft_consensus.cc:482] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:24:48.090044 20682 raft_consensus.cc:3037] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:48.091075 20682 raft_consensus.cc:504] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5158881a11941c8aeda1b110bb4a606" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 44421 } } peers { permanent_uuid: "81b4acfb70a94b3aa237417a50e618f0" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 45093 } } peers { permanent_uuid: "8d43a42d7b0141d98900a55c744c2b0f" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33775 } }
I20260926 20:24:48.091447 20682 leader_election.cc:288] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f [CANDIDATE]: Term 1 election: Requested vote from peers c5158881a11941c8aeda1b110bb4a606 (127.0.116.194:44421), 81b4acfb70a94b3aa237417a50e618f0 (127.0.116.195:45093)
I20260926 20:24:48.091851 20481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2338a83ad859401db65012695d7eb2b6" candidate_uuid: "8d43a42d7b0141d98900a55c744c2b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5158881a11941c8aeda1b110bb4a606"
I20260926 20:24:48.091843 20612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2338a83ad859401db65012695d7eb2b6" candidate_uuid: "8d43a42d7b0141d98900a55c744c2b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b4acfb70a94b3aa237417a50e618f0"
I20260926 20:24:48.091977 20481 raft_consensus.cc:3037] T 2338a83ad859401db65012695d7eb2b6 P c5158881a11941c8aeda1b110bb4a606 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:48.091977 20612 raft_consensus.cc:3037] T 2338a83ad859401db65012695d7eb2b6 P 81b4acfb70a94b3aa237417a50e618f0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:48.093304 20481 raft_consensus.cc:2454] T 2338a83ad859401db65012695d7eb2b6 P c5158881a11941c8aeda1b110bb4a606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d43a42d7b0141d98900a55c744c2b0f in term 1.
I20260926 20:24:48.093304 20612 raft_consensus.cc:2454] T 2338a83ad859401db65012695d7eb2b6 P 81b4acfb70a94b3aa237417a50e618f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d43a42d7b0141d98900a55c744c2b0f in term 1.
I20260926 20:24:48.093652 20283 leader_election.cc:302] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 81b4acfb70a94b3aa237417a50e618f0, 8d43a42d7b0141d98900a55c744c2b0f, c5158881a11941c8aeda1b110bb4a606; no voters:
I20260926 20:24:48.093852 20682 raft_consensus.cc:2781] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:24:48.094095 20682 raft_consensus.cc:686] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f [term 1 LEADER]: Becoming Leader. State: Replica: 8d43a42d7b0141d98900a55c744c2b0f, State: Running, Role: LEADER
I20260926 20:24:48.094429 20682 consensus_queue.cc:237] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5158881a11941c8aeda1b110bb4a606" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 44421 } } peers { permanent_uuid: "81b4acfb70a94b3aa237417a50e618f0" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 45093 } } peers { permanent_uuid: "8d43a42d7b0141d98900a55c744c2b0f" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33775 } }
I20260926 20:24:48.097757 20206 catalog_manager.cc:5495] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d43a42d7b0141d98900a55c744c2b0f (127.0.116.193). New cstate: current_term: 1 leader_uuid: "8d43a42d7b0141d98900a55c744c2b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5158881a11941c8aeda1b110bb4a606" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 44421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81b4acfb70a94b3aa237417a50e618f0" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 45093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d43a42d7b0141d98900a55c744c2b0f" member_type: VOTER last_known_addr { host: "127.0.116.193" port: 33775 } health_report { overall_health: HEALTHY } } }
I20260926 20:24:48.115121 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:48.111791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44152 (local address 127.0.116.193:33775)
0926 20:24:48.111908 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:48.111912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:48.111929 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:24:48.112089 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:48.112092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:48.112149 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:48.112238 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:24:48.112471 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:48.112984 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:48.113807 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:48.114026 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:48.114106 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:48.114368 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:24:48.114802 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:24:48.114891 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:48.114964 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 20:24:48.115011 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:24:48.121407 20612 raft_consensus.cc:1261] T 2338a83ad859401db65012695d7eb2b6 P 81b4acfb70a94b3aa237417a50e618f0 [term 1 FOLLOWER]: Refusing update from remote peer 8d43a42d7b0141d98900a55c744c2b0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:24:48.121407 20481 raft_consensus.cc:1261] T 2338a83ad859401db65012695d7eb2b6 P c5158881a11941c8aeda1b110bb4a606 [term 1 FOLLOWER]: Refusing update from remote peer 8d43a42d7b0141d98900a55c744c2b0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:24:48.122233 20686 consensus_queue.cc:1035] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5158881a11941c8aeda1b110bb4a606" member_type: VOTER last_known_addr { host: "127.0.116.194" port: 44421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:24:48.122395 20682 consensus_queue.cc:1035] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b4acfb70a94b3aa237417a50e618f0" member_type: VOTER last_known_addr { host: "127.0.116.195" port: 45093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:24:48.129757 20689 mvcc.cc:205] Tried to move back new op lower bound from 7333700764137508864 to 7333700764037267456. Current Snapshot: MvccSnapshot[applied={T|T < 7333700764137508864}]
I20260926 20:24:48.129900 20691 mvcc.cc:205] Tried to move back new op lower bound from 7333700764137508864 to 7333700764037267456. Current Snapshot: MvccSnapshot[applied={T|T < 7333700764137508864}]
I20260926 20:24:48.138185 20206 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:24:48.138368 20206 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:24:48.140832 20206 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34c9f77634ae4579acf1bd596122d2df: Sending DeleteTablet for 3 replicas of tablet 2338a83ad859401db65012695d7eb2b6
I20260926 20:24:48.141646 20461 tablet_service.cc:1514] Processing DeleteTablet for tablet 2338a83ad859401db65012695d7eb2b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:24:48 UTC) from {username='slave'} at 127.0.0.1:43204
I20260926 20:24:48.141885 20592 tablet_service.cc:1514] Processing DeleteTablet for tablet 2338a83ad859401db65012695d7eb2b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:24:48 UTC) from {username='slave'} at 127.0.0.1:39148
I20260926 20:24:48.142371 20700 tablet_replica.cc:330] T 2338a83ad859401db65012695d7eb2b6 P 81b4acfb70a94b3aa237417a50e618f0: stopping tablet replica
I20260926 20:24:48.142393 20329 tablet_service.cc:1514] Processing DeleteTablet for tablet 2338a83ad859401db65012695d7eb2b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:24:48 UTC) from {username='slave'} at 127.0.0.1:44138
I20260926 20:24:48.142725 20700 raft_consensus.cc:2229] T 2338a83ad859401db65012695d7eb2b6 P 81b4acfb70a94b3aa237417a50e618f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:24:48.142867 20701 tablet_replica.cc:330] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f: stopping tablet replica
I20260926 20:24:48.143174 20701 raft_consensus.cc:2229] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:24:48.143221 20700 raft_consensus.cc:2258] T 2338a83ad859401db65012695d7eb2b6 P 81b4acfb70a94b3aa237417a50e618f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:24:48.143431 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 20268
I20260926 20:24:48.143628 20701 raft_consensus.cc:2258] T 2338a83ad859401db65012695d7eb2b6 P 8d43a42d7b0141d98900a55c744c2b0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:24:48.145932 20700 ts_tablet_manager.cc:1905] T 2338a83ad859401db65012695d7eb2b6 P 81b4acfb70a94b3aa237417a50e618f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:24:48.146181 20699 tablet_replica.cc:330] T 2338a83ad859401db65012695d7eb2b6 P c5158881a11941c8aeda1b110bb4a606: stopping tablet replica
I20260926 20:24:48.146531 20699 raft_consensus.cc:2229] T 2338a83ad859401db65012695d7eb2b6 P c5158881a11941c8aeda1b110bb4a606 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:24:48.146860 20699 raft_consensus.cc:2258] T 2338a83ad859401db65012695d7eb2b6 P c5158881a11941c8aeda1b110bb4a606 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:24:48.148634 20699 ts_tablet_manager.cc:1905] T 2338a83ad859401db65012695d7eb2b6 P c5158881a11941c8aeda1b110bb4a606: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:24:48.148989 20700 ts_tablet_manager.cc:1918] T 2338a83ad859401db65012695d7eb2b6 P 81b4acfb70a94b3aa237417a50e618f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:24:48.149106 20700 log.cc:1196] T 2338a83ad859401db65012695d7eb2b6 P 81b4acfb70a94b3aa237417a50e618f0: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-2/wal/wals/2338a83ad859401db65012695d7eb2b6
I20260926 20:24:48.149423 20700 ts_tablet_manager.cc:1939] T 2338a83ad859401db65012695d7eb2b6 P 81b4acfb70a94b3aa237417a50e618f0: Deleting consensus metadata
I20260926 20:24:48.150823 20194 catalog_manager.cc:4841] TS 81b4acfb70a94b3aa237417a50e618f0 (127.0.116.195:45093): tablet 2338a83ad859401db65012695d7eb2b6 (table test-table [id=01da925a836a4ca8b7a91be0f128c97b]) successfully deleted
I20260926 20:24:48.152102 20699 ts_tablet_manager.cc:1918] T 2338a83ad859401db65012695d7eb2b6 P c5158881a11941c8aeda1b110bb4a606: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260926 20:24:48.152173 20193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.116.193:33775 (error 108)
I20260926 20:24:48.152231 20699 log.cc:1196] T 2338a83ad859401db65012695d7eb2b6 P c5158881a11941c8aeda1b110bb4a606: Deleting WAL directory at /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790454033974695-467-0/minicluster-data/ts-1/wal/wals/2338a83ad859401db65012695d7eb2b6
I20260926 20:24:48.152549 20699 ts_tablet_manager.cc:1939] T 2338a83ad859401db65012695d7eb2b6 P c5158881a11941c8aeda1b110bb4a606: Deleting consensus metadata
I20260926 20:24:48.153141 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 20399
I20260926 20:24:48.153280 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:48.152978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.193:33775 (local address 127.0.0.1:44160)
0926 20:24:48.153106 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:48.153171 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.193:33775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 20:24:48.153767 20193 catalog_manager.cc:4584] TS 8d43a42d7b0141d98900a55c744c2b0f (127.0.116.193:33775): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2338a83ad859401db65012695d7eb2b6: Network error: Client connection negotiation failed: client connection to 127.0.116.193:33775: connect: Connection refused (error 111)
I20260926 20:24:48.159242 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 20530
I20260926 20:24:48.159848 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:48.159474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.194:44421 (local address 127.0.0.1:43218)
0926 20:24:48.159691 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 20:24:48.159752 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.194:44421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260926 20:24:48.164659 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 20177
2026-09-26T20:24:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1172 ms)
[----------] 6 tests from EncryptionPolicyTest (11113 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:24:48 dist-test-slave-qsm9 krb5kdc[20705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:24:48 dist-test-slave-qsm9 krb5kdc[20705](info): set up 2 sockets
Sep 26 20:24:48 dist-test-slave-qsm9 krb5kdc[20705](info): commencing operation
krb5kdc: starting...
W20260926 20:24:49.268194 467 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:24:49 dist-test-slave-qsm9 krb5kdc[20705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:24:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:24:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:24:49.399274 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35683
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:24:49.514487 20718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:49.514823 20718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:49.514868 20718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:49.519109 20718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:24:49.519193 20718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:49.519214 20718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:49.519232 20718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:24:49.519248 20718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:24:49.524926 20718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35683
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:49.526147 20718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:49.527235 20718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:24:49.536120 20718 server_base.cc:1034] running on GCE node
W20260926 20:24:49.536062 20723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:49.536049 20724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:49.536067 20726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:49.536873 20718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:49.538084 20718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:49.539301 20718 hybrid_clock.cc:648] HybridClock initialized: now 1790454289539279 us; error 28 us; skew 500 ppm
Sep 26 20:24:49 dist-test-slave-qsm9 krb5kdc[20705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:49.542279 20718 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:24:49.543226 20718 webserver.cc:460] Webserver started at http://127.0.0.1:42921/ using document root <none> and password file <none>
I20260926 20:24:49.543900 20718 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:49.543998 20718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:49.544224 20718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:49.546257 20718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "10b778ec5e23496188536c45f3d7fa00"
format_stamp: "Formatted at 2026-09-26 20:24:49 on dist-test-slave-qsm9"
server_key: "450d31366dbef3084228a2109b544849"
server_key_iv: "a052011944758140d48a86077085b6ce"
server_key_version: "encryptionkey@0"
I20260926 20:24:49.546834 20718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "10b778ec5e23496188536c45f3d7fa00"
format_stamp: "Formatted at 2026-09-26 20:24:49 on dist-test-slave-qsm9"
server_key: "450d31366dbef3084228a2109b544849"
server_key_iv: "a052011944758140d48a86077085b6ce"
server_key_version: "encryptionkey@0"
I20260926 20:24:49.550652 20718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:24:49.553243 20733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:49.554587 20718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:24:49.554755 20718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "10b778ec5e23496188536c45f3d7fa00"
format_stamp: "Formatted at 2026-09-26 20:24:49 on dist-test-slave-qsm9"
server_key: "450d31366dbef3084228a2109b544849"
server_key_iv: "a052011944758140d48a86077085b6ce"
server_key_version: "encryptionkey@0"
I20260926 20:24:49.554944 20718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:49.574095 20718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:49.577602 20718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:49.577824 20718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:49.585402 20718 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35683
I20260926 20:24:49.585425 20785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35683 every 8 connection(s)
I20260926 20:24:49.586437 20718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:24:49.586907 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 20718
I20260926 20:24:49.587076 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:24:49.587431 467 external_mini_cluster.cc:1428] Setting key 6f271b1c4794d9226802883ab17e6263
I20260926 20:24:49.589744 20786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:49.595266 20786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00: Bootstrap starting.
Sep 26 20:24:49 dist-test-slave-qsm9 krb5kdc[20705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:24:49.597664 20786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:49.598423 20786 log.cc:824] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:49.600463 20786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00: No bootstrap required, opened a new log
I20260926 20:24:49.601910 20789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:49.589027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39187 (local address 127.0.0.1:35683)
0926 20:24:49.589488 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:49.589503 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:49.589539 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:24:49.590269 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:49.590288 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:49.590670 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:49.591006 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:49.591897 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:49.592884 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:49.593730 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:49.594204 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:49.594388 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:49.597024 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:49.597029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:49.597042 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:49.597072 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:49.598978 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:49.599532 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:49.599538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:49.599549 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:49.599660 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:49.600007 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:49.600010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:49.600012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:49.600414 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:49.600633 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:49.600954 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 20:24:49.601179 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":152,"threads_started":1}
I20260926 20:24:49.604688 20786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10b778ec5e23496188536c45f3d7fa00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35683 } }
I20260926 20:24:49.604997 20786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:49.605058 20786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b778ec5e23496188536c45f3d7fa00, State: Initialized, Role: FOLLOWER
I20260926 20:24:49.605405 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:24:49.605582 20786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10b778ec5e23496188536c45f3d7fa00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35683 } }
I20260926 20:24:49.605739 20786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:24:49.605820 20786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:24:49.605928 20786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:49.606985 20786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10b778ec5e23496188536c45f3d7fa00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35683 } }
I20260926 20:24:49.607370 20786 leader_election.cc:302] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10b778ec5e23496188536c45f3d7fa00; no voters:
I20260926 20:24:49.607761 20786 leader_election.cc:288] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:24:49.607864 20795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:24:49.608254 20795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00 [term 1 LEADER]: Becoming Leader. State: Replica: 10b778ec5e23496188536c45f3d7fa00, State: Running, Role: LEADER
I20260926 20:24:49.608631 20795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10b778ec5e23496188536c45f3d7fa00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35683 } }
I20260926 20:24:49.609249 20786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:24:49.610546 20797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10b778ec5e23496188536c45f3d7fa00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10b778ec5e23496188536c45f3d7fa00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35683 } } }
I20260926 20:24:49.610692 20797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00 [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:49.611631 20798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10b778ec5e23496188536c45f3d7fa00. Latest consensus state: current_term: 1 leader_uuid: "10b778ec5e23496188536c45f3d7fa00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10b778ec5e23496188536c45f3d7fa00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35683 } } }
I20260926 20:24:49.611975 20798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00 [sys.catalog]: This master's current role is: LEADER
W20260926 20:24:49.614785 20811 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:24:49.614885 20811 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:24:49.615034 20806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:24:49.615900 20806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:24:49.619755 20789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:49.607875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37767 (local address 127.0.0.1:35683)
0926 20:24:49.608013 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:49.608018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:49.610285 (+ 2267us) server_negotiation.cc:407] Connection header received
0926 20:24:49.610510 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:49.610513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:49.610561 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:49.610611 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:49.611714 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:49.612380 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:49.613359 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:49.613707 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:49.613834 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:49.615648 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:49.615653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:49.615655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:49.615681 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:49.617534 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:49.618188 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:49.618191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:49.618192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:49.618300 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:49.618691 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:49.618702 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:49.618703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:49.618906 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:49.619073 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:49.619388 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 20:24:49.619533 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:24:49.622854 20806 catalog_manager.cc:1347] Generated new cluster ID: eee1167fc3354413b4b4a1b6e583c1da
I20260926 20:24:49.622918 20806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:24:49.646643 20806 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:24:49.647784 20806 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:24:49.654374 20806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10b778ec5e23496188536c45f3d7fa00: Generated new TSK 0
I20260926 20:24:49.655150 20806 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:24:49.664440 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 20718
W20260926 20:24:49.669975 20703 connection.cc:535] client connection to 127.0.0.1:35683 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35683): Connection reset by peer (error 104)
2026-09-26T20:24:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1498 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:24:49 dist-test-slave-qsm9 krb5kdc[20817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:24:49 dist-test-slave-qsm9 krb5kdc[20817](info): set up 2 sockets
Sep 26 20:24:49 dist-test-slave-qsm9 krb5kdc[20817](info): commencing operation
krb5kdc: starting...
W20260926 20:24:50.708225 467 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:24:50 dist-test-slave-qsm9 krb5kdc[20817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:24:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:24:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:24:50.835005 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:24:50.950049 20830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:50.950402 20830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:50.950461 20830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:50.954496 20830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:24:50.954569 20830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:50.954589 20830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:50.954605 20830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:24:50.954620 20830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:24:50.960093 20830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42845
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:50.961277 20830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:50.962297 20830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:50.970988 20838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:50.971009 20836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:50.971004 20835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:50.971153 20830 server_base.cc:1034] running on GCE node
I20260926 20:24:50.971953 20830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:50.973115 20830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:50.974296 20830 hybrid_clock.cc:648] HybridClock initialized: now 1790454290974281 us; error 32 us; skew 500 ppm
Sep 26 20:24:50 dist-test-slave-qsm9 krb5kdc[20817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:50.977283 20830 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:24:50.978267 20830 webserver.cc:460] Webserver started at http://127.0.0.1:40555/ using document root <none> and password file <none>
I20260926 20:24:50.978837 20830 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:50.978933 20830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:50.979156 20830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:50.981371 20830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "e0a0b021f8cb4a41a945a725afb68415"
format_stamp: "Formatted at 2026-09-26 20:24:50 on dist-test-slave-qsm9"
server_key: "e3674ef8ed979c8988688db1b2b8c3bf"
server_key_iv: "07b69f4e68c9f02dbdf3e053fae17487"
server_key_version: "encryptionkey@0"
I20260926 20:24:50.981954 20830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "e0a0b021f8cb4a41a945a725afb68415"
format_stamp: "Formatted at 2026-09-26 20:24:50 on dist-test-slave-qsm9"
server_key: "e3674ef8ed979c8988688db1b2b8c3bf"
server_key_iv: "07b69f4e68c9f02dbdf3e053fae17487"
server_key_version: "encryptionkey@0"
I20260926 20:24:50.985740 20830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:24:50.988363 20845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:50.989635 20830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:24:50.989775 20830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "e0a0b021f8cb4a41a945a725afb68415"
format_stamp: "Formatted at 2026-09-26 20:24:50 on dist-test-slave-qsm9"
server_key: "e3674ef8ed979c8988688db1b2b8c3bf"
server_key_iv: "07b69f4e68c9f02dbdf3e053fae17487"
server_key_version: "encryptionkey@0"
I20260926 20:24:50.989898 20830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:51.014016 20830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:51.017776 20830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:51.018019 20830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:51.025437 20830 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42845
I20260926 20:24:51.025454 20897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42845 every 8 connection(s)
I20260926 20:24:51.026429 20830 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:24:51.029254 20898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:51.031910 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 20830
I20260926 20:24:51.032083 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:24:51.032411 467 external_mini_cluster.cc:1428] Setting key c94d64d2c7bdb6a3a242a79b9892e995
I20260926 20:24:51.035398 20898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415: Bootstrap starting.
I20260926 20:24:51.037937 20898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:51.038858 20898 log.cc:824] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:51.040993 20898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415: No bootstrap required, opened a new log
I20260926 20:24:51.042388 20901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:51.033996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60269 (local address 127.0.0.1:42845)
0926 20:24:51.034494 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:51.034512 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:51.034551 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:24:51.035227 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:51.035245 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:51.035740 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:51.036064 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:51.036420 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:51.037532 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:51.038808 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:51.039358 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:51.039585 (+ 227us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:51.040521 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:51.040526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:51.040539 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:51.040982 (+ 443us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:51.041037 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:51.041061 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:51.041202 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:51.041441 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 20:24:51.041591 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":168,"threads_started":1}
I20260926 20:24:51.044905 20898 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a0b021f8cb4a41a945a725afb68415" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42845 } }
I20260926 20:24:51.045245 20898 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:51.045328 20898 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0a0b021f8cb4a41a945a725afb68415, State: Initialized, Role: FOLLOWER
I20260926 20:24:51.045631 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:24:51.045835 20898 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a0b021f8cb4a41a945a725afb68415" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42845 } }
I20260926 20:24:51.045996 20898 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:24:51.046059 20898 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:24:51.046167 20898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:51.047544 20898 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a0b021f8cb4a41a945a725afb68415" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42845 } }
I20260926 20:24:51.047936 20898 leader_election.cc:302] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0a0b021f8cb4a41a945a725afb68415; no voters:
I20260926 20:24:51.048321 20898 leader_election.cc:288] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:24:51.048441 20908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:24:51.048696 20908 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415 [term 1 LEADER]: Becoming Leader. State: Replica: e0a0b021f8cb4a41a945a725afb68415, State: Running, Role: LEADER
I20260926 20:24:51.049093 20908 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a0b021f8cb4a41a945a725afb68415" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42845 } }
I20260926 20:24:51.049757 20898 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:24:51.050647 20910 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0a0b021f8cb4a41a945a725afb68415" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a0b021f8cb4a41a945a725afb68415" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42845 } } }
I20260926 20:24:51.050808 20910 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415 [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:51.051060 20910 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0a0b021f8cb4a41a945a725afb68415. Latest consensus state: current_term: 1 leader_uuid: "e0a0b021f8cb4a41a945a725afb68415" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0a0b021f8cb4a41a945a725afb68415" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42845 } } }
I20260926 20:24:51.051172 20910 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415 [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:51.053386 20901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:51.048021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56721 (local address 127.0.0.1:42845)
0926 20:24:51.048140 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:51.048143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:51.048450 (+ 307us) server_negotiation.cc:407] Connection header received
0926 20:24:51.048640 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:51.048643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:51.048694 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:51.048772 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:51.049173 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:51.049927 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:51.051157 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:51.051574 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:51.051652 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:51.052973 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:51.052976 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:51.052978 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:51.052990 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:51.053015 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:51.053025 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:51.053096 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:51.053234 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 20:24:51.053264 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 20:24:51.054832 20923 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:24:51.054898 20923 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:24:51.054975 20917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:24:51.055769 20917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:24:51.061527 20917 catalog_manager.cc:1347] Generated new cluster ID: 31a5de6894b24355bf608b14d025ff79
I20260926 20:24:51.061609 20917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:24:51.075309 20917 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:24:51.076741 20917 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:24:51.084329 20917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0a0b021f8cb4a41a945a725afb68415: Generated new TSK 0
I20260926 20:24:51.085157 20917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:24:51.094365 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 20830
W20260926 20:24:51.099982 20815 connection.cc:535] client connection to 127.0.0.1:42845 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42845): Connection reset by peer (error 104)
2026-09-26T20:24:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:24:51 dist-test-slave-qsm9 krb5kdc[20929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:24:51 dist-test-slave-qsm9 krb5kdc[20929](info): set up 2 sockets
Sep 26 20:24:51 dist-test-slave-qsm9 krb5kdc[20929](info): commencing operation
krb5kdc: starting...
W20260926 20:24:52.132081 467 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:24:52 dist-test-slave-qsm9 krb5kdc[20929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:24:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:24:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:24:52.259039 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:24:52.371356 20942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:52.371687 20942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:52.371733 20942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:52.375800 20942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:24:52.375864 20942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:52.375883 20942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:52.375900 20942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:24:52.375916 20942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:24:52.381433 20942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33315
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:52.382490 20942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:52.383458 20942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:52.392025 20948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:52.392045 20947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:52.392058 20950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:52.392351 20942 server_base.cc:1034] running on GCE node
I20260926 20:24:52.393105 20942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:52.394292 20942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:52.395572 20942 hybrid_clock.cc:648] HybridClock initialized: now 1790454292395560 us; error 34 us; skew 500 ppm
Sep 26 20:24:52 dist-test-slave-qsm9 krb5kdc[20929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:52.404724 20942 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:24:52.405833 20942 webserver.cc:460] Webserver started at http://127.0.0.1:38577/ using document root <none> and password file <none>
I20260926 20:24:52.406445 20942 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:52.406540 20942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:52.406773 20942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:52.408963 20942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "6323c8b950614f7dba750bdaa1fcd4eb"
format_stamp: "Formatted at 2026-09-26 20:24:52 on dist-test-slave-qsm9"
server_key: "37d2f6f37d40a262dc1685c9f1bfdb34"
server_key_iv: "d4ad76954752ce3279ef0328eb5efb6b"
server_key_version: "encryptionkey@0"
I20260926 20:24:52.409548 20942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "6323c8b950614f7dba750bdaa1fcd4eb"
format_stamp: "Formatted at 2026-09-26 20:24:52 on dist-test-slave-qsm9"
server_key: "37d2f6f37d40a262dc1685c9f1bfdb34"
server_key_iv: "d4ad76954752ce3279ef0328eb5efb6b"
server_key_version: "encryptionkey@0"
I20260926 20:24:52.413471 20942 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260926 20:24:52.416239 20957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:52.417689 20942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:24:52.417897 20942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "6323c8b950614f7dba750bdaa1fcd4eb"
format_stamp: "Formatted at 2026-09-26 20:24:52 on dist-test-slave-qsm9"
server_key: "37d2f6f37d40a262dc1685c9f1bfdb34"
server_key_iv: "d4ad76954752ce3279ef0328eb5efb6b"
server_key_version: "encryptionkey@0"
I20260926 20:24:52.418035 20942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:52.422887 20942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:52.426059 20942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:52.426259 20942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:52.433593 20942 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33315
I20260926 20:24:52.433597 21009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33315 every 8 connection(s)
I20260926 20:24:52.434712 20942 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:24:52.435567 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 20942
I20260926 20:24:52.435721 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:24:52.436025 467 external_mini_cluster.cc:1428] Setting key 1df8dcd9576a8848f63cafe3db95f11e
I20260926 20:24:52.438083 21010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:52.441527 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:52.437482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53941 (local address 127.0.0.1:33315)
0926 20:24:52.437928 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:52.437945 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:52.437983 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:24:52.438717 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:52.438733 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:52.439029 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:52.439328 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:52.440221 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:52.440227 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:52.440239 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:52.440250 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:52.440313 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:52.440336 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:52.440462 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:52.440572 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 20:24:52.440743 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":144,"threads_started":1}
I20260926 20:24:52.444088 21010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb: Bootstrap starting.
I20260926 20:24:52.444422 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:24:52.446800 21010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:52.447775 21010 log.cc:824] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:52.448552 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:52.446724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41031 (local address 127.0.0.1:33315)
0926 20:24:52.446863 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:52.446867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:52.447140 (+ 273us) server_negotiation.cc:407] Connection header received
0926 20:24:52.447262 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:52.447264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:52.447296 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:52.447359 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:52.448184 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:52.448187 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:52.448190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:52.448201 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:52.448226 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:52.448242 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:52.448309 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:52.448425 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 20:24:52.448456 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:24:52.450212 21010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb: No bootstrap required, opened a new log
I20260926 20:24:52.453881 21010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6323c8b950614f7dba750bdaa1fcd4eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33315 } }
I20260926 20:24:52.454176 21010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:52.454262 21010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6323c8b950614f7dba750bdaa1fcd4eb, State: Initialized, Role: FOLLOWER
I20260926 20:24:52.454743 21010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6323c8b950614f7dba750bdaa1fcd4eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33315 } }
I20260926 20:24:52.454896 21010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:24:52.454979 21010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:24:52.455081 21010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:52.456147 21010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6323c8b950614f7dba750bdaa1fcd4eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33315 } }
I20260926 20:24:52.456508 21010 leader_election.cc:302] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6323c8b950614f7dba750bdaa1fcd4eb; no voters:
I20260926 20:24:52.456835 21010 leader_election.cc:288] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:24:52.456950 21020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:24:52.457187 21020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb [term 1 LEADER]: Becoming Leader. State: Replica: 6323c8b950614f7dba750bdaa1fcd4eb, State: Running, Role: LEADER
I20260926 20:24:52.457594 21020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6323c8b950614f7dba750bdaa1fcd4eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33315 } }
I20260926 20:24:52.458138 21010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:24:52.459407 21021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6323c8b950614f7dba750bdaa1fcd4eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6323c8b950614f7dba750bdaa1fcd4eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33315 } } }
I20260926 20:24:52.459415 21022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6323c8b950614f7dba750bdaa1fcd4eb. Latest consensus state: current_term: 1 leader_uuid: "6323c8b950614f7dba750bdaa1fcd4eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6323c8b950614f7dba750bdaa1fcd4eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33315 } } }
I20260926 20:24:52.459566 21021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:52.459574 21022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb [sys.catalog]: This master's current role is: LEADER
W20260926 20:24:52.463155 21035 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:24:52.463229 21035 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:24:52.463308 21029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:24:52.464066 21029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:24:52.470218 21029 catalog_manager.cc:1347] Generated new cluster ID: 9b0335c063324194ba1f1ff488cf4046
I20260926 20:24:52.470302 21029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:24:52.492035 21029 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:24:52.492425 21029 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:24:52.503017 21029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6323c8b950614f7dba750bdaa1fcd4eb: Generated new TSK 0
I20260926 20:24:52.503890 21029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:24:52.514663 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 20942
2026-09-26T20:24:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1419 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:24:52 dist-test-slave-qsm9 krb5kdc[21041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:24:52 dist-test-slave-qsm9 krb5kdc[21041](info): set up 2 sockets
Sep 26 20:24:52 dist-test-slave-qsm9 krb5kdc[21041](info): commencing operation
krb5kdc: starting...
W20260926 20:24:53.556200 467 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:24:53 dist-test-slave-qsm9 krb5kdc[21041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:24:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:24:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:24:53.692082 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38847
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:24:53.813238 21054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:53.813611 21054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:53.813695 21054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:53.818229 21054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:24:53.818328 21054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:53.818367 21054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:53.818405 21054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:24:53.818441 21054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:24:53.824267 21054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:53.825471 21054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:53.826579 21054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:53.836228 21059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:53.836216 21060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:53.836216 21062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:53.836427 21054 server_base.cc:1034] running on GCE node
I20260926 20:24:53.837217 21054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:53.838551 21054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:53.839712 21054 hybrid_clock.cc:648] HybridClock initialized: now 1790454293839693 us; error 35 us; skew 500 ppm
Sep 26 20:24:53 dist-test-slave-qsm9 krb5kdc[21041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:53.842753 21054 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:24:53.843801 21054 webserver.cc:460] Webserver started at http://127.0.0.1:45027/ using document root <none> and password file <none>
I20260926 20:24:53.844414 21054 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:53.844516 21054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:53.844753 21054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:53.846854 21054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "17f92cb473984c28909362acf9272d2e"
format_stamp: "Formatted at 2026-09-26 20:24:53 on dist-test-slave-qsm9"
server_key: "05ab440cd0e2dd60e9d739841711bba2"
server_key_iv: "869f9a647f1a2c8537a5e79ae610567a"
server_key_version: "encryptionkey@0"
I20260926 20:24:53.847440 21054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "17f92cb473984c28909362acf9272d2e"
format_stamp: "Formatted at 2026-09-26 20:24:53 on dist-test-slave-qsm9"
server_key: "05ab440cd0e2dd60e9d739841711bba2"
server_key_iv: "869f9a647f1a2c8537a5e79ae610567a"
server_key_version: "encryptionkey@0"
I20260926 20:24:53.851478 21054 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 20:24:53.854331 21069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:53.855947 21054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 20:24:53.856115 21054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "17f92cb473984c28909362acf9272d2e"
format_stamp: "Formatted at 2026-09-26 20:24:53 on dist-test-slave-qsm9"
server_key: "05ab440cd0e2dd60e9d739841711bba2"
server_key_iv: "869f9a647f1a2c8537a5e79ae610567a"
server_key_version: "encryptionkey@0"
I20260926 20:24:53.856254 21054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:53.875466 21054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:53.879192 21054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:53.879400 21054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:53.887182 21054 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38847
I20260926 20:24:53.887212 21121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38847 every 8 connection(s)
I20260926 20:24:53.888317 21054 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:24:53.889187 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 21054
I20260926 20:24:53.889349 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:24:53.889657 467 external_mini_cluster.cc:1428] Setting key 2f816e26fac8f74ac3fd13ae3d3b9188
I20260926 20:24:53.891987 21122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:53.898813 21122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e: Bootstrap starting.
Sep 26 20:24:53 dist-test-slave-qsm9 krb5kdc[21041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:24:53.901752 21122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:53.902858 21122 log.cc:824] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:53.905779 21122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e: No bootstrap required, opened a new log
I20260926 20:24:53.907284 21125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:53.891126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50449 (local address 127.0.0.1:38847)
0926 20:24:53.891666 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:53.891684 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:53.891724 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:24:53.892526 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:53.892548 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:53.892929 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:53.893306 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:53.894426 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:53.895636 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:53.896987 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:53.897500 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:53.897639 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:53.901070 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:53.901076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:53.901092 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:53.901131 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:53.903669 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:53.904290 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:53.904297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:53.904305 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:53.904468 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:53.904845 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:53.904848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:53.904850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:53.905420 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:53.905610 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:53.905903 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 20:24:53.906254 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":153,"threads_started":1}
I20260926 20:24:53.910959 21122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f92cb473984c28909362acf9272d2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38847 } }
I20260926 20:24:53.911317 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:24:53.911305 21122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:53.911372 21122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f92cb473984c28909362acf9272d2e, State: Initialized, Role: FOLLOWER
I20260926 20:24:53.911945 21122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f92cb473984c28909362acf9272d2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38847 } }
I20260926 20:24:53.912117 21122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:24:53.912179 21122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:24:53.912302 21122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:53.913681 21122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f92cb473984c28909362acf9272d2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38847 } }
I20260926 20:24:53.914113 21122 leader_election.cc:302] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17f92cb473984c28909362acf9272d2e; no voters:
I20260926 20:24:53.914486 21122 leader_election.cc:288] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:24:53.914584 21132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:24:53.914834 21132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e [term 1 LEADER]: Becoming Leader. State: Replica: 17f92cb473984c28909362acf9272d2e, State: Running, Role: LEADER
I20260926 20:24:53.915252 21132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f92cb473984c28909362acf9272d2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38847 } }
I20260926 20:24:53.916019 21122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:24:53.916963 21134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17f92cb473984c28909362acf9272d2e. Latest consensus state: current_term: 1 leader_uuid: "17f92cb473984c28909362acf9272d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f92cb473984c28909362acf9272d2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38847 } } }
I20260926 20:24:53.917364 21134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:53.917246 21133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17f92cb473984c28909362acf9272d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f92cb473984c28909362acf9272d2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38847 } } }
I20260926 20:24:53.917692 21133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e [sys.catalog]: This master's current role is: LEADER
W20260926 20:24:53.922003 21148 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:24:53.922137 21148 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:24:53.922232 21141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:24:53.923192 21141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:24:53.923341 21125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:53.913817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41179 (local address 127.0.0.1:38847)
0926 20:24:53.913941 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:53.913946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:53.914102 (+ 156us) server_negotiation.cc:407] Connection header received
0926 20:24:53.914208 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:53.914213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:53.914271 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:53.914362 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:53.915268 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:53.916074 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:53.917322 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:53.917802 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:53.917940 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:53.919722 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:53.919726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:53.919730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:53.919757 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:53.921380 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:53.921907 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:53.921911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:53.921913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:53.922006 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:53.922322 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:53.922325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:53.922327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:53.922568 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:53.922714 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:53.922990 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 20:24:53.923125 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:24:53.929417 21141 catalog_manager.cc:1347] Generated new cluster ID: 7801b080545f462da97e53f5c0fc6faf
I20260926 20:24:53.929500 21141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:24:53.959661 21141 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:24:53.961066 21141 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:24:53.971228 21141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17f92cb473984c28909362acf9272d2e: Generated new TSK 0
I20260926 20:24:53.972179 21141 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:24:53.984382 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 21054
2026-09-26T20:24:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1470 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:24:54 dist-test-slave-qsm9 krb5kdc[21153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:24:54 dist-test-slave-qsm9 krb5kdc[21153](info): set up 2 sockets
Sep 26 20:24:54 dist-test-slave-qsm9 krb5kdc[21153](info): commencing operation
krb5kdc: starting...
W20260926 20:24:55.032078 467 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:24:55 dist-test-slave-qsm9 krb5kdc[21153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:24:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:24:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:24:55.161003 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:24:55.273792 21166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:55.274097 21166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:55.274142 21166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:55.278348 21166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:24:55.278421 21166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:55.278441 21166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:55.278460 21166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:24:55.278474 21166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:24:55.283903 21166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:55.284974 21166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:55.285971 21166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:55.294648 21171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:55.294638 21172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:55.294691 21166 server_base.cc:1034] running on GCE node
W20260926 20:24:55.294638 21174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:55.295603 21166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:55.296842 21166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:55.298050 21166 hybrid_clock.cc:648] HybridClock initialized: now 1790454295298029 us; error 31 us; skew 500 ppm
Sep 26 20:24:55 dist-test-slave-qsm9 krb5kdc[21153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:55.301393 21166 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:24:55.302388 21166 webserver.cc:460] Webserver started at http://127.0.0.1:32897/ using document root <none> and password file <none>
I20260926 20:24:55.302953 21166 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:55.303016 21166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:55.303184 21166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:55.305429 21166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "21430cee81c4452e8a5c5432f113b48f"
format_stamp: "Formatted at 2026-09-26 20:24:55 on dist-test-slave-qsm9"
server_key: "e3f094f801818147306825795839d00f"
server_key_iv: "8e6334258526d6104bdfdb3c6cc21d39"
server_key_version: "encryptionkey@0"
I20260926 20:24:55.305986 21166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "21430cee81c4452e8a5c5432f113b48f"
format_stamp: "Formatted at 2026-09-26 20:24:55 on dist-test-slave-qsm9"
server_key: "e3f094f801818147306825795839d00f"
server_key_iv: "8e6334258526d6104bdfdb3c6cc21d39"
server_key_version: "encryptionkey@0"
I20260926 20:24:55.310009 21166 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 20:24:55.312738 21181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:55.314332 21166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:24:55.314474 21166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "21430cee81c4452e8a5c5432f113b48f"
format_stamp: "Formatted at 2026-09-26 20:24:55 on dist-test-slave-qsm9"
server_key: "e3f094f801818147306825795839d00f"
server_key_iv: "8e6334258526d6104bdfdb3c6cc21d39"
server_key_version: "encryptionkey@0"
I20260926 20:24:55.314639 21166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:55.343439 21166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:55.346762 21166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:55.346926 21166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:55.353884 21166 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39217
I20260926 20:24:55.354017 21233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39217 every 8 connection(s)
I20260926 20:24:55.354905 21166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:24:55.357749 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 21166
I20260926 20:24:55.357860 21234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:55.357915 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:24:55.358196 467 external_mini_cluster.cc:1428] Setting key c9dabed22babab6d1a420f537213fa25
I20260926 20:24:55.364605 21234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f: Bootstrap starting.
I20260926 20:24:55.367115 21234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:55.367165 21237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:55.359626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46035 (local address 127.0.0.1:39217)
0926 20:24:55.360082 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:55.360097 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:55.360132 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:24:55.360767 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:55.360786 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:55.361142 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:55.361520 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:55.361923 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:55.362906 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:55.363873 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:55.364355 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:55.364539 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:55.365427 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:55.365431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:55.365445 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:55.365860 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:55.365916 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:55.365939 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:55.366051 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:55.366214 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 20:24:55.366427 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":146,"threads_started":1}
I20260926 20:24:55.368117 21234 log.cc:824] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:55.370369 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:24:55.370549 21234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f: No bootstrap required, opened a new log
I20260926 20:24:55.374536 21234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21430cee81c4452e8a5c5432f113b48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39217 } }
I20260926 20:24:55.374785 21234 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:55.374871 21234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21430cee81c4452e8a5c5432f113b48f, State: Initialized, Role: FOLLOWER
I20260926 20:24:55.375372 21234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21430cee81c4452e8a5c5432f113b48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39217 } }
I20260926 20:24:55.375568 21234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:24:55.375646 21234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:24:55.375775 21234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:55.376471 21237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:55.372560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59093 (local address 127.0.0.1:39217)
0926 20:24:55.372677 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:55.372681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:55.372869 (+ 188us) server_negotiation.cc:407] Connection header received
0926 20:24:55.373031 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:55.373034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:55.373084 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:55.373163 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:55.373421 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:55.373923 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:55.374725 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:55.375043 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:55.375151 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:55.376046 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:55.376050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:55.376052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:55.376065 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:55.376097 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:55.376108 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:55.376179 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:55.376272 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 20:24:55.376338 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:24:55.376991 21234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21430cee81c4452e8a5c5432f113b48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39217 } }
I20260926 20:24:55.377379 21234 leader_election.cc:302] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21430cee81c4452e8a5c5432f113b48f; no voters:
I20260926 20:24:55.377780 21234 leader_election.cc:288] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:24:55.377859 21244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:24:55.378119 21244 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f [term 1 LEADER]: Becoming Leader. State: Replica: 21430cee81c4452e8a5c5432f113b48f, State: Running, Role: LEADER
I20260926 20:24:55.378468 21244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21430cee81c4452e8a5c5432f113b48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39217 } }
I20260926 20:24:55.379067 21234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:24:55.380287 21246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21430cee81c4452e8a5c5432f113b48f. Latest consensus state: current_term: 1 leader_uuid: "21430cee81c4452e8a5c5432f113b48f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21430cee81c4452e8a5c5432f113b48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39217 } } }
I20260926 20:24:55.380416 21246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:55.380679 21245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21430cee81c4452e8a5c5432f113b48f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21430cee81c4452e8a5c5432f113b48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39217 } } }
I20260926 20:24:55.380761 21245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:55.380798 21253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:24:55.384050 21253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:24:55.389338 21253 catalog_manager.cc:1347] Generated new cluster ID: afc1dd9973cc4fed9656158e38814f4d
I20260926 20:24:55.389406 21253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:24:55.412024 21253 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:24:55.413375 21253 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:24:55.420449 21253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21430cee81c4452e8a5c5432f113b48f: Generated new TSK 0
I20260926 20:24:55.421310 21253 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:24:55.429658 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 21166
2026-09-26T20:24:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:24:55 dist-test-slave-qsm9 krb5kdc[21265](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:24:55 dist-test-slave-qsm9 krb5kdc[21265](info): set up 2 sockets
Sep 26 20:24:55 dist-test-slave-qsm9 krb5kdc[21265](info): commencing operation
krb5kdc: starting...
W20260926 20:24:56.464119 467 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:24:56 dist-test-slave-qsm9 krb5kdc[21265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:24:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:24:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:24:56.590826 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34261
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:24:56.710426 21278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:56.710758 21278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:56.710893 21278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:56.715250 21278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:24:56.715361 21278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:56.715385 21278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:56.715406 21278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:24:56.715425 21278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:24:56.721426 21278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:56.722651 21278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:56.723805 21278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:56.733064 21284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:56.733064 21286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:56.733143 21283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:56.733546 21278 server_base.cc:1034] running on GCE node
I20260926 20:24:56.734316 21278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:56.735630 21278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:56.736809 21278 hybrid_clock.cc:648] HybridClock initialized: now 1790454296736779 us; error 49 us; skew 500 ppm
Sep 26 20:24:56 dist-test-slave-qsm9 krb5kdc[21265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:56.739851 21278 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:24:56.740886 21278 webserver.cc:460] Webserver started at http://127.0.0.1:40135/ using document root <none> and password file <none>
I20260926 20:24:56.741513 21278 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:56.741611 21278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:56.741866 21278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:56.744241 21278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "3050b79542504043baa60923be242645"
format_stamp: "Formatted at 2026-09-26 20:24:56 on dist-test-slave-qsm9"
server_key: "11fed12ad57c22b357932ab48ed09dd4"
server_key_iv: "50fea1df4a18c0cdbf4c48ba4c110a9e"
server_key_version: "encryptionkey@0"
I20260926 20:24:56.744884 21278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "3050b79542504043baa60923be242645"
format_stamp: "Formatted at 2026-09-26 20:24:56 on dist-test-slave-qsm9"
server_key: "11fed12ad57c22b357932ab48ed09dd4"
server_key_iv: "50fea1df4a18c0cdbf4c48ba4c110a9e"
server_key_version: "encryptionkey@0"
I20260926 20:24:56.749727 21278 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:24:56.752662 21293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:56.754065 21278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:24:56.754246 21278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "3050b79542504043baa60923be242645"
format_stamp: "Formatted at 2026-09-26 20:24:56 on dist-test-slave-qsm9"
server_key: "11fed12ad57c22b357932ab48ed09dd4"
server_key_iv: "50fea1df4a18c0cdbf4c48ba4c110a9e"
server_key_version: "encryptionkey@0"
I20260926 20:24:56.754401 21278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:56.760018 21278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:56.763255 21278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:56.763484 21278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:56.770761 21278 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34261
I20260926 20:24:56.770780 21345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34261 every 8 connection(s)
I20260926 20:24:56.771838 21278 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:24:56.774693 21346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:56.777611 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 21278
I20260926 20:24:56.777789 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:24:56.778128 467 external_mini_cluster.cc:1428] Setting key 3bd4fb00ff5608997db9009ea4fab7fe
I20260926 20:24:56.781967 21346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645: Bootstrap starting.
I20260926 20:24:56.784418 21349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:56.779815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56273 (local address 127.0.0.1:34261)
0926 20:24:56.780405 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:56.780423 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:56.780477 (+ 54us) server_negotiation.cc:407] Connection header received
0926 20:24:56.781247 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:56.781270 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:56.781613 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:56.782114 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:56.783032 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:56.783037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:56.783052 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:56.783063 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:56.783122 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:56.783151 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:56.783319 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:56.783407 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 20:24:56.783625 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":191,"threads_started":1}
I20260926 20:24:56.784916 21346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:56.786154 21346 log.cc:824] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:56.787905 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:24:56.788995 21346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645: No bootstrap required, opened a new log
I20260926 20:24:56.792250 21349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:56.790383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34455 (local address 127.0.0.1:34261)
0926 20:24:56.790513 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:56.790518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:56.790660 (+ 142us) server_negotiation.cc:407] Connection header received
0926 20:24:56.790772 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:56.790775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:56.790811 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:56.790893 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:56.791731 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:56.791735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:56.791738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:56.791753 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:56.791783 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:56.791798 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:56.791891 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:56.792064 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 20:24:56.792105 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:24:56.793521 21346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3050b79542504043baa60923be242645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34261 } }
I20260926 20:24:56.793754 21346 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:56.793823 21346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3050b79542504043baa60923be242645, State: Initialized, Role: FOLLOWER
I20260926 20:24:56.794305 21346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3050b79542504043baa60923be242645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34261 } }
I20260926 20:24:56.794438 21346 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:24:56.794481 21346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:24:56.794577 21346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:56.795634 21346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3050b79542504043baa60923be242645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34261 } }
I20260926 20:24:56.795989 21346 leader_election.cc:302] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3050b79542504043baa60923be242645; no voters:
I20260926 20:24:56.796330 21346 leader_election.cc:288] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:24:56.796406 21356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:24:56.796676 21356 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645 [term 1 LEADER]: Becoming Leader. State: Replica: 3050b79542504043baa60923be242645, State: Running, Role: LEADER
I20260926 20:24:56.797032 21356 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3050b79542504043baa60923be242645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34261 } }
I20260926 20:24:56.797703 21346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:24:56.799357 21358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3050b79542504043baa60923be242645. Latest consensus state: current_term: 1 leader_uuid: "3050b79542504043baa60923be242645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3050b79542504043baa60923be242645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34261 } } }
I20260926 20:24:56.799429 21357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3050b79542504043baa60923be242645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3050b79542504043baa60923be242645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34261 } } }
I20260926 20:24:56.799643 21357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645 [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:56.799918 21358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645 [sys.catalog]: This master's current role is: LEADER
W20260926 20:24:56.802815 21371 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:24:56.802915 21371 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:24:56.802986 21370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:24:56.803798 21370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:24:56.809528 21370 catalog_manager.cc:1347] Generated new cluster ID: 947be8c796b94cc68f89dd1a6ef8230c
I20260926 20:24:56.809602 21370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:24:56.824972 21370 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:24:56.825343 21370 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:24:56.832034 21370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3050b79542504043baa60923be242645: Generated new TSK 0
I20260926 20:24:56.832774 21370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:24:56.838634 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 21278
2026-09-26T20:24:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1408 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:24:56 dist-test-slave-qsm9 krb5kdc[21377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:24:56 dist-test-slave-qsm9 krb5kdc[21377](info): set up 2 sockets
Sep 26 20:24:56 dist-test-slave-qsm9 krb5kdc[21377](info): commencing operation
krb5kdc: starting...
W20260926 20:24:57.936244 467 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:24:58 dist-test-slave-qsm9 krb5kdc[21377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:24:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:24:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:24:58.063906 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34837
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:24:58.175982 21390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:58.176270 21390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:58.176313 21390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:58.180503 21390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:24:58.180564 21390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:58.180584 21390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:58.180601 21390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:24:58.180639 21390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:24:58.186110 21390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:58.187247 21390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:58.188380 21390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:58.197422 21395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:58.197507 21398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:58.197466 21396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:58.197561 21390 server_base.cc:1034] running on GCE node
I20260926 20:24:58.198324 21390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:58.199431 21390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:58.200603 21390 hybrid_clock.cc:648] HybridClock initialized: now 1790454298200582 us; error 32 us; skew 500 ppm
Sep 26 20:24:58 dist-test-slave-qsm9 krb5kdc[21377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:58.210310 21390 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:24:58.211318 21390 webserver.cc:460] Webserver started at http://127.0.0.1:40687/ using document root <none> and password file <none>
I20260926 20:24:58.212026 21390 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:58.212129 21390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:58.212381 21390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:58.214581 21390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "441111b14cec4b0cb12ef4bf027d46eb"
format_stamp: "Formatted at 2026-09-26 20:24:58 on dist-test-slave-qsm9"
server_key: "da1ecc3cf37ad0acacbca203a2b688b6"
server_key_iv: "22f9207719ed349a90b2258f2e50085b"
server_key_version: "encryptionkey@0"
I20260926 20:24:58.215188 21390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "441111b14cec4b0cb12ef4bf027d46eb"
format_stamp: "Formatted at 2026-09-26 20:24:58 on dist-test-slave-qsm9"
server_key: "da1ecc3cf37ad0acacbca203a2b688b6"
server_key_iv: "22f9207719ed349a90b2258f2e50085b"
server_key_version: "encryptionkey@0"
I20260926 20:24:58.219153 21390 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 20:24:58.222013 21405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:58.223261 21390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 20:24:58.223412 21390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "441111b14cec4b0cb12ef4bf027d46eb"
format_stamp: "Formatted at 2026-09-26 20:24:58 on dist-test-slave-qsm9"
server_key: "da1ecc3cf37ad0acacbca203a2b688b6"
server_key_iv: "22f9207719ed349a90b2258f2e50085b"
server_key_version: "encryptionkey@0"
I20260926 20:24:58.223591 21390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:58.281317 21390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:58.284759 21390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:58.284997 21390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:58.292186 21390 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34837
I20260926 20:24:58.292199 21457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34837 every 8 connection(s)
I20260926 20:24:58.293166 21390 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:24:58.296015 21458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:58.301318 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 21390
I20260926 20:24:58.301470 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:24:58.301750 467 external_mini_cluster.cc:1428] Setting key f034e616d950fa8686968829889ca29c
I20260926 20:24:58.301831 21458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb: Bootstrap starting.
I20260926 20:24:58.304350 21458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:58.305229 21458 log.cc:824] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:58.307349 21458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb: No bootstrap required, opened a new log
Sep 26 20:24:58 dist-test-slave-qsm9 krb5kdc[21377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:24:58.311782 21458 raft_consensus.cc:348] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441111b14cec4b0cb12ef4bf027d46eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34837 } }
I20260926 20:24:58.312000 21458 raft_consensus.cc:374] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:58.312055 21458 raft_consensus.cc:729] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441111b14cec4b0cb12ef4bf027d46eb, State: Initialized, Role: FOLLOWER
I20260926 20:24:58.312597 21458 consensus_queue.cc:260] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441111b14cec4b0cb12ef4bf027d46eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34837 } }
I20260926 20:24:58.312731 21458 raft_consensus.cc:388] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:24:58.312824 21458 raft_consensus.cc:482] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:24:58.312942 21458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:58.313820 21458 raft_consensus.cc:504] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441111b14cec4b0cb12ef4bf027d46eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34837 } }
I20260926 20:24:58.314162 21458 leader_election.cc:302] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 441111b14cec4b0cb12ef4bf027d46eb; no voters:
I20260926 20:24:58.314488 21458 leader_election.cc:288] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:24:58.314553 21463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:24:58.314749 21463 raft_consensus.cc:686] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb [term 1 LEADER]: Becoming Leader. State: Replica: 441111b14cec4b0cb12ef4bf027d46eb, State: Running, Role: LEADER
I20260926 20:24:58.315104 21463 consensus_queue.cc:237] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441111b14cec4b0cb12ef4bf027d46eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34837 } }
I20260926 20:24:58.315809 21458 sys_catalog.cc:564] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:24:58.317389 21461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:58.303060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:47919 (local address 127.0.0.1:34837)
0926 20:24:58.303465 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:58.303480 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:58.303540 (+ 60us) server_negotiation.cc:407] Connection header received
0926 20:24:58.304144 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:58.304160 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:58.304571 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:58.304860 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:58.305860 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:58.306981 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:58.308260 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:58.308744 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:58.308864 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:58.311877 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:58.311922 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:58.311935 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:58.311962 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:58.313757 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:58.314279 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:58.314286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:58.314292 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:58.314419 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:58.314773 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:58.314780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:58.314782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:58.315264 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:58.315533 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:58.316150 (+ 617us) server_negotiation.cc:299] Negotiation successful
0926 20:24:58.316434 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":119,"threads_started":1}
I20260926 20:24:58.318462 21464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "441111b14cec4b0cb12ef4bf027d46eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441111b14cec4b0cb12ef4bf027d46eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34837 } } }
I20260926 20:24:58.318584 21464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:58.318926 21465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 441111b14cec4b0cb12ef4bf027d46eb. Latest consensus state: current_term: 1 leader_uuid: "441111b14cec4b0cb12ef4bf027d46eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441111b14cec4b0cb12ef4bf027d46eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34837 } } }
I20260926 20:24:58.319046 21465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:58.320849 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 20:24:58.321310 21478 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:24:58.321398 21478 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:24:58.321566 21479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:24:58.322366 21479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:24:58.328730 21479 catalog_manager.cc:1347] Generated new cluster ID: 4af7613d8fe243b3a2a1897632b0dad9
I20260926 20:24:58.328845 21479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:24:58.331939 21461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:58.323246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:58489 (local address 127.0.0.1:34837)
0926 20:24:58.323369 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:58.323374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:58.323844 (+ 470us) server_negotiation.cc:407] Connection header received
0926 20:24:58.323970 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:58.323973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:58.324021 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:58.324116 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:58.324867 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:58.325397 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:58.326223 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:58.326563 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:58.326667 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:58.328580 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:58.328584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:58.328586 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:24:58.328612 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:58.330163 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:58.330651 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:58.330654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:58.330656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:58.330737 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:24:58.331085 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:24:58.331088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:24:58.331090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:24:58.331272 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:58.331431 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:58.331685 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 20:24:58.331778 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:24:58.363009 21479 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:24:58.364089 21479 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:24:58.372979 21479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 441111b14cec4b0cb12ef4bf027d46eb: Generated new TSK 0
I20260926 20:24:58.373719 21479 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:24:58.381062 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 21390
W20260926 20:24:58.386750 21375 connection.cc:535] client connection to 127.0.0.1:34837 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34837): Connection reset by peer (error 104)
2026-09-26T20:24:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1542 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:24:58 dist-test-slave-qsm9 krb5kdc[21489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:24:58 dist-test-slave-qsm9 krb5kdc[21489](info): set up 2 sockets
Sep 26 20:24:58 dist-test-slave-qsm9 krb5kdc[21489](info): commencing operation
krb5kdc: starting...
W20260926 20:24:59.424113 467 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:24:59 dist-test-slave-qsm9 krb5kdc[21489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:24:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:24:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:24:59.554373 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:24:59.669227 21502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:24:59.669534 21502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:24:59.669579 21502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:24:59.673964 21502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:24:59.674135 21502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:24:59.674211 21502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:24:59.674252 21502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:24:59.674281 21502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:24:59.683274 21502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:24:59.684536 21502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:24:59.685662 21502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:24:59.694723 21510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:59.694748 21508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:24:59.694773 21507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:24:59.694985 21502 server_base.cc:1034] running on GCE node
I20260926 20:24:59.695747 21502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:24:59.696964 21502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:24:59.698132 21502 hybrid_clock.cc:648] HybridClock initialized: now 1790454299698120 us; error 27 us; skew 500 ppm
Sep 26 20:24:59 dist-test-slave-qsm9 krb5kdc[21489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:24:59.701023 21502 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:24:59.702037 21502 webserver.cc:460] Webserver started at http://127.0.0.1:38207/ using document root <none> and password file <none>
I20260926 20:24:59.702627 21502 fs_manager.cc:359] Metadata directory not provided
I20260926 20:24:59.702719 21502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:24:59.702962 21502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:24:59.705142 21502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "e689080f97d94cf08342a5958d0611fc"
format_stamp: "Formatted at 2026-09-26 20:24:59 on dist-test-slave-qsm9"
server_key: "4463c91b79213ba41ec5941bd7bab65b"
server_key_iv: "bcd8c0b644c9d62f2ec28eddd5bfa0da"
server_key_version: "encryptionkey@0"
I20260926 20:24:59.705747 21502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "e689080f97d94cf08342a5958d0611fc"
format_stamp: "Formatted at 2026-09-26 20:24:59 on dist-test-slave-qsm9"
server_key: "4463c91b79213ba41ec5941bd7bab65b"
server_key_iv: "bcd8c0b644c9d62f2ec28eddd5bfa0da"
server_key_version: "encryptionkey@0"
I20260926 20:24:59.709750 21502 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260926 20:24:59.712499 21517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:24:59.713853 21502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:24:59.714015 21502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "e689080f97d94cf08342a5958d0611fc"
format_stamp: "Formatted at 2026-09-26 20:24:59 on dist-test-slave-qsm9"
server_key: "4463c91b79213ba41ec5941bd7bab65b"
server_key_iv: "bcd8c0b644c9d62f2ec28eddd5bfa0da"
server_key_version: "encryptionkey@0"
I20260926 20:24:59.714154 21502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:24:59.736317 21502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:24:59.739823 21502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:24:59.740044 21502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:24:59.747879 21502 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44977
I20260926 20:24:59.747907 21569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44977 every 8 connection(s)
I20260926 20:24:59.748924 21502 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:24:59.751402 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 21502
I20260926 20:24:59.751598 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:24:59.751915 467 external_mini_cluster.cc:1428] Setting key 6e49e331530b118e34efbe31fd909c71
I20260926 20:24:59.752049 21570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:24:59.759792 21570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc: Bootstrap starting.
I20260926 20:24:59.762794 21570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc: Neither blocks nor log segments found. Creating new log.
I20260926 20:24:59.762863 21573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:59.753461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:59975 (local address 127.0.0.1:44977)
0926 20:24:59.753912 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:59.753931 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:59.753976 (+ 45us) server_negotiation.cc:407] Connection header received
0926 20:24:59.754953 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:59.754982 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:59.755517 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:59.755953 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:59.756396 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:59.757858 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:59.758841 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:59.759565 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:59.759738 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:59.760716 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:59.760723 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:59.760740 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:59.761271 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:59.761344 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:59.761376 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:59.761526 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:59.761712 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 20:24:59.761859 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":126,"threads_started":1}
I20260926 20:24:59.763756 21570 log.cc:824] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc: Log is configured to *not* fsync() on all Append() calls
I20260926 20:24:59.766077 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:24:59.766152 21570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc: No bootstrap required, opened a new log
I20260926 20:24:59.770732 21570 raft_consensus.cc:348] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e689080f97d94cf08342a5958d0611fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44977 } }
I20260926 20:24:59.771057 21570 raft_consensus.cc:374] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:24:59.771135 21570 raft_consensus.cc:729] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e689080f97d94cf08342a5958d0611fc, State: Initialized, Role: FOLLOWER
I20260926 20:24:59.771672 21570 consensus_queue.cc:260] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e689080f97d94cf08342a5958d0611fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44977 } }
I20260926 20:24:59.771816 21570 raft_consensus.cc:388] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:24:59.771884 21570 raft_consensus.cc:482] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:24:59.771988 21570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:24:59.772531 21573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:24:59.768173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:38391 (local address 127.0.0.1:44977)
0926 20:24:59.768290 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 20:24:59.768293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:24:59.768523 (+ 230us) server_negotiation.cc:407] Connection header received
0926 20:24:59.768627 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:24:59.768632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:24:59.768693 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:24:59.768781 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:24:59.769020 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:59.769785 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:59.770577 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:24:59.770898 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:24:59.771004 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:24:59.772099 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:24:59.772103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:24:59.772105 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:24:59.772120 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:24:59.772149 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:24:59.772165 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:24:59.772264 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 20:24:59.772359 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 20:24:59.772394 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:24:59.773196 21570 raft_consensus.cc:504] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e689080f97d94cf08342a5958d0611fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44977 } }
I20260926 20:24:59.773564 21570 leader_election.cc:302] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e689080f97d94cf08342a5958d0611fc; no voters:
I20260926 20:24:59.773900 21570 leader_election.cc:288] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:24:59.774021 21580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:24:59.774253 21580 raft_consensus.cc:686] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc [term 1 LEADER]: Becoming Leader. State: Replica: e689080f97d94cf08342a5958d0611fc, State: Running, Role: LEADER
I20260926 20:24:59.774576 21580 consensus_queue.cc:237] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e689080f97d94cf08342a5958d0611fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44977 } }
I20260926 20:24:59.775223 21570 sys_catalog.cc:564] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:24:59.776607 21581 sys_catalog.cc:455] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e689080f97d94cf08342a5958d0611fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e689080f97d94cf08342a5958d0611fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44977 } } }
I20260926 20:24:59.776619 21582 sys_catalog.cc:455] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e689080f97d94cf08342a5958d0611fc. Latest consensus state: current_term: 1 leader_uuid: "e689080f97d94cf08342a5958d0611fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e689080f97d94cf08342a5958d0611fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44977 } } }
I20260926 20:24:59.776733 21581 sys_catalog.cc:458] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:59.776759 21582 sys_catalog.cc:458] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc [sys.catalog]: This master's current role is: LEADER
I20260926 20:24:59.777616 21589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:24:59.780512 21589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:24:59.785907 21589 catalog_manager.cc:1347] Generated new cluster ID: 83424d959e8948fa8dc728cdc2a1b4a3
I20260926 20:24:59.785969 21589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:24:59.795807 21589 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:24:59.796772 21589 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:24:59.805058 21589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e689080f97d94cf08342a5958d0611fc: Generated new TSK 0
I20260926 20:24:59.805791 21589 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:24:59.817278 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 21502
W20260926 20:24:59.823114 21487 connection.cc:535] client connection to 127.0.0.1:44977 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44977): Connection reset by peer (error 104)
2026-09-26T20:24:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:24:59 dist-test-slave-qsm9 krb5kdc[21601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:24:59 dist-test-slave-qsm9 krb5kdc[21601](info): set up 2 sockets
Sep 26 20:24:59 dist-test-slave-qsm9 krb5kdc[21601](info): commencing operation
krb5kdc: starting...
W20260926 20:25:00.860153 467 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:25:00 dist-test-slave-qsm9 krb5kdc[21601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:25:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:25:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:25:00.995325 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41383
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:25:01.110046 21614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:25:01.110356 21614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:25:01.110404 21614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:25:01.114684 21614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:25:01.114766 21614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:25:01.114786 21614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:25:01.114804 21614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:25:01.114820 21614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:25:01.120307 21614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:25:01.121500 21614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:25:01.122541 21614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:25:01.131546 21620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:01.131561 21622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:01.131671 21614 server_base.cc:1034] running on GCE node
W20260926 20:25:01.131546 21619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:01.132655 21614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:25:01.134027 21614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:25:01.135216 21614 hybrid_clock.cc:648] HybridClock initialized: now 1790454301135187 us; error 42 us; skew 500 ppm
Sep 26 20:25:01 dist-test-slave-qsm9 krb5kdc[21601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:25:01.138371 21614 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:25:01.139371 21614 webserver.cc:460] Webserver started at http://127.0.0.1:37593/ using document root <none> and password file <none>
I20260926 20:25:01.140004 21614 fs_manager.cc:359] Metadata directory not provided
I20260926 20:25:01.140071 21614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:25:01.140244 21614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:25:01.142412 21614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "32aa8cac423c4bfab2d788b5ec2210be"
format_stamp: "Formatted at 2026-09-26 20:25:01 on dist-test-slave-qsm9"
server_key: "73f29d64b5e2ff30fa2865fb5828241f"
server_key_iv: "5e8741e93adc6a7d1872990ba7a789ab"
server_key_version: "encryptionkey@0"
I20260926 20:25:01.143184 21614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "32aa8cac423c4bfab2d788b5ec2210be"
format_stamp: "Formatted at 2026-09-26 20:25:01 on dist-test-slave-qsm9"
server_key: "73f29d64b5e2ff30fa2865fb5828241f"
server_key_iv: "5e8741e93adc6a7d1872990ba7a789ab"
server_key_version: "encryptionkey@0"
I20260926 20:25:01.147207 21614 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 20:25:01.149791 21629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:25:01.151153 21614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:25:01.151273 21614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "32aa8cac423c4bfab2d788b5ec2210be"
format_stamp: "Formatted at 2026-09-26 20:25:01 on dist-test-slave-qsm9"
server_key: "73f29d64b5e2ff30fa2865fb5828241f"
server_key_iv: "5e8741e93adc6a7d1872990ba7a789ab"
server_key_version: "encryptionkey@0"
I20260926 20:25:01.151376 21614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:25:01.156179 21614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:25:01.159381 21614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:25:01.159629 21614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:25:01.166879 21614 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41383
I20260926 20:25:01.166962 21681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41383 every 8 connection(s)
I20260926 20:25:01.167991 21614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:25:01.171049 21682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:25:01.172137 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 21614
I20260926 20:25:01.172289 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:25:01.172586 467 external_mini_cluster.cc:1428] Setting key 59d8b74e9fc8d51ad0024fd172020e35
I20260926 20:25:01.178275 21682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be: Bootstrap starting.
I20260926 20:25:01.178514 21685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:01.174069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:42877 (local address 127.0.0.1:41383)
0926 20:25:01.174517 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:01.174538 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:01.174587 (+ 49us) server_negotiation.cc:407] Connection header received
0926 20:25:01.175441 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:01.175466 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:01.175870 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:01.176297 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:01.177064 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:01.177070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:01.177085 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:25:01.177097 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:01.177164 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:25:01.177194 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:01.177336 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:01.177406 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 20:25:01.177570 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":111,"threads_started":1}
I20260926 20:25:01.181041 21682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be: Neither blocks nor log segments found. Creating new log.
I20260926 20:25:01.181972 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:25:01.182024 21682 log.cc:824] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be: Log is configured to *not* fsync() on all Append() calls
I20260926 20:25:01.184614 21682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be: No bootstrap required, opened a new log
I20260926 20:25:01.186357 21685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:01.184374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:44247 (local address 127.0.0.1:41383)
0926 20:25:01.184533 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:01.184539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:01.184805 (+ 266us) server_negotiation.cc:407] Connection header received
0926 20:25:01.184864 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:01.184867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:01.184921 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:01.185006 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:01.185784 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:01.185787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:01.185789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:25:01.185800 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:01.185830 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:25:01.185844 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:01.185931 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:01.186084 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 20:25:01.186170 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:25:01.189091 21682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa8cac423c4bfab2d788b5ec2210be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41383 } }
I20260926 20:25:01.189354 21682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:25:01.189440 21682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32aa8cac423c4bfab2d788b5ec2210be, State: Initialized, Role: FOLLOWER
I20260926 20:25:01.189913 21682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa8cac423c4bfab2d788b5ec2210be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41383 } }
I20260926 20:25:01.190076 21682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:25:01.190167 21682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:25:01.190299 21682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:25:01.191387 21682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa8cac423c4bfab2d788b5ec2210be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41383 } }
I20260926 20:25:01.191805 21682 leader_election.cc:302] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32aa8cac423c4bfab2d788b5ec2210be; no voters:
I20260926 20:25:01.192155 21682 leader_election.cc:288] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:25:01.192297 21692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:25:01.192572 21692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be [term 1 LEADER]: Becoming Leader. State: Replica: 32aa8cac423c4bfab2d788b5ec2210be, State: Running, Role: LEADER
I20260926 20:25:01.192911 21692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa8cac423c4bfab2d788b5ec2210be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41383 } }
I20260926 20:25:01.193570 21682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:25:01.194810 21694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32aa8cac423c4bfab2d788b5ec2210be. Latest consensus state: current_term: 1 leader_uuid: "32aa8cac423c4bfab2d788b5ec2210be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa8cac423c4bfab2d788b5ec2210be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41383 } } }
I20260926 20:25:01.194810 21693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32aa8cac423c4bfab2d788b5ec2210be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa8cac423c4bfab2d788b5ec2210be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41383 } } }
I20260926 20:25:01.195014 21693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:01.195288 21694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be [sys.catalog]: This master's current role is: LEADER
W20260926 20:25:01.198603 21707 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:25:01.198686 21707 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:25:01.198770 21701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:25:01.199628 21701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:25:01.205716 21701 catalog_manager.cc:1347] Generated new cluster ID: 1181344e6e6e4696a21dc89f5ae01418
I20260926 20:25:01.205806 21701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:25:01.219614 21701 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:25:01.220041 21701 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:25:01.225762 21701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32aa8cac423c4bfab2d788b5ec2210be: Generated new TSK 0
I20260926 20:25:01.226670 21701 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:25:01.234829 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 21614
2026-09-26T20:25:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:25:01 dist-test-slave-qsm9 krb5kdc[21713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:25:01 dist-test-slave-qsm9 krb5kdc[21713](info): set up 2 sockets
Sep 26 20:25:01 dist-test-slave-qsm9 krb5kdc[21713](info): commencing operation
krb5kdc: starting...
W20260926 20:25:02.284227 467 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:25:02 dist-test-slave-qsm9 krb5kdc[21713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:25:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:25:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:25:02.415664 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:25:02.530642 21726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:25:02.530985 21726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:25:02.531062 21726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:25:02.535349 21726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:25:02.535463 21726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:25:02.535502 21726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:25:02.535593 21726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:25:02.535652 21726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:25:02.541427 21726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36371
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:25:02.542644 21726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:25:02.543771 21726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:25:02.552650 21734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:02.552650 21732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:02.552672 21731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:02.552769 21726 server_base.cc:1034] running on GCE node
I20260926 20:25:02.553583 21726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:25:02.554737 21726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:25:02.555938 21726 hybrid_clock.cc:648] HybridClock initialized: now 1790454302555924 us; error 30 us; skew 500 ppm
Sep 26 20:25:02 dist-test-slave-qsm9 krb5kdc[21713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:25:02.558784 21726 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:25:02.559859 21726 webserver.cc:460] Webserver started at http://127.0.0.1:33941/ using document root <none> and password file <none>
I20260926 20:25:02.560449 21726 fs_manager.cc:359] Metadata directory not provided
I20260926 20:25:02.560542 21726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:25:02.560783 21726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:25:02.562918 21726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "5cf622116a06447d8658930d5c8e2077"
format_stamp: "Formatted at 2026-09-26 20:25:02 on dist-test-slave-qsm9"
server_key: "5838d51eaad9c01faf726703fef8f0e7"
server_key_iv: "e2ea6df2af7cffe2c38c6981766be32b"
server_key_version: "encryptionkey@0"
I20260926 20:25:02.563484 21726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "5cf622116a06447d8658930d5c8e2077"
format_stamp: "Formatted at 2026-09-26 20:25:02 on dist-test-slave-qsm9"
server_key: "5838d51eaad9c01faf726703fef8f0e7"
server_key_iv: "e2ea6df2af7cffe2c38c6981766be32b"
server_key_version: "encryptionkey@0"
I20260926 20:25:02.567346 21726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:25:02.570237 21741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:25:02.571714 21726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:25:02.571872 21726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "5cf622116a06447d8658930d5c8e2077"
format_stamp: "Formatted at 2026-09-26 20:25:02 on dist-test-slave-qsm9"
server_key: "5838d51eaad9c01faf726703fef8f0e7"
server_key_iv: "e2ea6df2af7cffe2c38c6981766be32b"
server_key_version: "encryptionkey@0"
I20260926 20:25:02.572007 21726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:25:02.592594 21726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:25:02.596669 21726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:25:02.596940 21726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:25:02.605331 21726 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36371
I20260926 20:25:02.605342 21793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36371 every 8 connection(s)
I20260926 20:25:02.606426 21726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:25:02.609576 21794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:25:02.613335 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 21726
I20260926 20:25:02.613499 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:25:02.613833 467 external_mini_cluster.cc:1428] Setting key 7212ff3480f3ea3585584d29d4d2dacd
I20260926 20:25:02.616588 21794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077: Bootstrap starting.
I20260926 20:25:02.619011 21794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077: Neither blocks nor log segments found. Creating new log.
I20260926 20:25:02.619917 21794 log.cc:824] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077: Log is configured to *not* fsync() on all Append() calls
I20260926 20:25:02.622092 21794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077: No bootstrap required, opened a new log
Sep 26 20:25:02 dist-test-slave-qsm9 krb5kdc[21713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:25:02.625828 21794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf622116a06447d8658930d5c8e2077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36371 } }
I20260926 20:25:02.626111 21794 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:25:02.626212 21794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cf622116a06447d8658930d5c8e2077, State: Initialized, Role: FOLLOWER
I20260926 20:25:02.626761 21794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf622116a06447d8658930d5c8e2077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36371 } }
I20260926 20:25:02.626942 21794 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:25:02.627040 21794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:25:02.627152 21794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:25:02.628268 21794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf622116a06447d8658930d5c8e2077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36371 } }
I20260926 20:25:02.628713 21794 leader_election.cc:302] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cf622116a06447d8658930d5c8e2077; no voters:
I20260926 20:25:02.629113 21794 leader_election.cc:288] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:25:02.629231 21799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:25:02.629470 21799 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077 [term 1 LEADER]: Becoming Leader. State: Replica: 5cf622116a06447d8658930d5c8e2077, State: Running, Role: LEADER
I20260926 20:25:02.629848 21797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:02.615452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:48829 (local address 127.0.0.1:36371)
0926 20:25:02.615962 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:02.615977 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:02.616014 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:25:02.616644 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:02.616662 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:02.617143 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:02.617503 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:02.618338 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:02.619393 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:02.620334 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:02.620809 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:02.621006 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:02.623743 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:02.623748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:02.623762 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:02.623792 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:02.626290 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:02.626862 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:02.626868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:02.626875 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:02.627056 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:02.627461 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:02.627465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:02.627467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:02.627944 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:02.628197 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:02.628485 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 20:25:02.628728 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":153,"threads_started":1}
I20260926 20:25:02.629845 21799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf622116a06447d8658930d5c8e2077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36371 } }
I20260926 20:25:02.630584 21794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:25:02.632158 21801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cf622116a06447d8658930d5c8e2077. Latest consensus state: current_term: 1 leader_uuid: "5cf622116a06447d8658930d5c8e2077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf622116a06447d8658930d5c8e2077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36371 } } }
I20260926 20:25:02.632274 21801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077 [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:02.632210 21800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cf622116a06447d8658930d5c8e2077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf622116a06447d8658930d5c8e2077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36371 } } }
I20260926 20:25:02.632330 21800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077 [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:02.633750 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 20:25:02.636631 21814 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:25:02.636751 21814 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:25:02.636843 21809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:25:02.637786 21809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:25:02.645314 21809 catalog_manager.cc:1347] Generated new cluster ID: 7662033f3c7c40128ba585987dd87712
I20260926 20:25:02.645398 21809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:25:02.646049 21797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:02.635950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:47213 (local address 127.0.0.1:36371)
0926 20:25:02.636055 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:02.636059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:02.636332 (+ 273us) server_negotiation.cc:407] Connection header received
0926 20:25:02.636471 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:02.636475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:02.636529 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:02.636623 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:02.637470 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:02.638205 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:02.639320 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:02.639687 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:02.639833 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:02.641701 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:02.641705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:02.641708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:02.641739 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:02.643686 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:02.644279 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:02.644283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:02.644285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:02.644378 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:02.644754 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:02.644757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:02.644760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:02.644977 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:02.645147 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:02.645778 (+ 631us) server_negotiation.cc:299] Negotiation successful
0926 20:25:02.645889 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:25:02.668305 21809 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:25:02.669358 21809 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:25:02.675700 21809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cf622116a06447d8658930d5c8e2077: Generated new TSK 0
I20260926 20:25:02.676523 21809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:25:02.684574 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 21726
W20260926 20:25:02.690106 21711 connection.cc:535] client connection to 127.0.0.1:36371 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36371): Connection reset by peer (error 104)
2026-09-26T20:25:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1449 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:25:02 dist-test-slave-qsm9 krb5kdc[21825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:25:02 dist-test-slave-qsm9 krb5kdc[21825](info): set up 2 sockets
Sep 26 20:25:02 dist-test-slave-qsm9 krb5kdc[21825](info): commencing operation
krb5kdc: starting...
W20260926 20:25:03.736097 467 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:25:03 dist-test-slave-qsm9 krb5kdc[21825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:25:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:25:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:25:03.863703 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36415
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:25:03.981559 21838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:25:03.981860 21838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:25:03.981940 21838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:25:03.986059 21838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:25:03.986135 21838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:25:03.986155 21838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:25:03.986173 21838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:25:03.986189 21838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:25:03.991873 21838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36415
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:25:03.993046 21838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:25:03.994135 21838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:25:04.003126 21846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:04.003129 21843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:04.003129 21844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:04.003710 21838 server_base.cc:1034] running on GCE node
I20260926 20:25:04.004343 21838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:25:04.005579 21838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:25:04.006803 21838 hybrid_clock.cc:648] HybridClock initialized: now 1790454304006770 us; error 49 us; skew 500 ppm
Sep 26 20:25:04 dist-test-slave-qsm9 krb5kdc[21825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:25:04.016439 21838 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:25:04.017450 21838 webserver.cc:460] Webserver started at http://127.0.0.1:42505/ using document root <none> and password file <none>
I20260926 20:25:04.018036 21838 fs_manager.cc:359] Metadata directory not provided
I20260926 20:25:04.018101 21838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:25:04.018285 21838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:25:04.020711 21838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "3705e97eaaea4abba1008c4b9b55f475"
format_stamp: "Formatted at 2026-09-26 20:25:04 on dist-test-slave-qsm9"
server_key: "ef58ff0fbdcc3aff4c900be99b2d6ffe"
server_key_iv: "b95546fa99a2ebea0fda4ea1efb5988e"
server_key_version: "encryptionkey@0"
I20260926 20:25:04.021289 21838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "3705e97eaaea4abba1008c4b9b55f475"
format_stamp: "Formatted at 2026-09-26 20:25:04 on dist-test-slave-qsm9"
server_key: "ef58ff0fbdcc3aff4c900be99b2d6ffe"
server_key_iv: "b95546fa99a2ebea0fda4ea1efb5988e"
server_key_version: "encryptionkey@0"
I20260926 20:25:04.025542 21838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:25:04.028285 21853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:25:04.029664 21838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:25:04.029817 21838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "3705e97eaaea4abba1008c4b9b55f475"
format_stamp: "Formatted at 2026-09-26 20:25:04 on dist-test-slave-qsm9"
server_key: "ef58ff0fbdcc3aff4c900be99b2d6ffe"
server_key_iv: "b95546fa99a2ebea0fda4ea1efb5988e"
server_key_version: "encryptionkey@0"
I20260926 20:25:04.030014 21838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:25:04.046437 21838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:25:04.049944 21838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:25:04.050158 21838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:25:04.057240 21838 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36415
I20260926 20:25:04.057262 21905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36415 every 8 connection(s)
I20260926 20:25:04.058239 21838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:25:04.061151 21906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:25:04.061175 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 21838
I20260926 20:25:04.061340 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:25:04.061676 467 external_mini_cluster.cc:1428] Setting key c572d52597e610d566ba21c3b10745d4
I20260926 20:25:04.068600 21906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475: Bootstrap starting.
I20260926 20:25:04.071574 21906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475: Neither blocks nor log segments found. Creating new log.
I20260926 20:25:04.072348 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:04.063207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:44905 (local address 127.0.0.1:36415)
0926 20:25:04.063681 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:04.063700 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:04.063744 (+ 44us) server_negotiation.cc:407] Connection header received
0926 20:25:04.064602 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:04.064628 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:04.065091 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:04.065487 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:04.065899 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:04.067415 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:04.068405 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:04.069082 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:04.069241 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:04.070231 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:04.070238 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:04.070255 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:25:04.070755 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:04.070824 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:25:04.070855 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:04.070982 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:04.071148 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 20:25:04.071288 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":129,"threads_started":1}
I20260926 20:25:04.072492 21906 log.cc:824] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475: Log is configured to *not* fsync() on all Append() calls
I20260926 20:25:04.075037 21906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475: No bootstrap required, opened a new log
I20260926 20:25:04.075956 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:25:04.079282 21906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3705e97eaaea4abba1008c4b9b55f475" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36415 } }
I20260926 20:25:04.079614 21906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:25:04.079694 21906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3705e97eaaea4abba1008c4b9b55f475, State: Initialized, Role: FOLLOWER
I20260926 20:25:04.080222 21906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3705e97eaaea4abba1008c4b9b55f475" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36415 } }
I20260926 20:25:04.080399 21906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:25:04.080482 21906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:25:04.080597 21906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:25:04.081791 21906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3705e97eaaea4abba1008c4b9b55f475" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36415 } }
I20260926 20:25:04.082186 21906 leader_election.cc:302] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3705e97eaaea4abba1008c4b9b55f475; no voters:
I20260926 20:25:04.082571 21906 leader_election.cc:288] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:25:04.082644 21916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:25:04.082854 21916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475 [term 1 LEADER]: Becoming Leader. State: Replica: 3705e97eaaea4abba1008c4b9b55f475, State: Running, Role: LEADER
I20260926 20:25:04.083590 21916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3705e97eaaea4abba1008c4b9b55f475" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36415 } }
I20260926 20:25:04.084060 21906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:25:04.084336 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:04.078146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:53517 (local address 127.0.0.1:36415)
0926 20:25:04.078270 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:04.078274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:04.079832 (+ 1558us) server_negotiation.cc:407] Connection header received
0926 20:25:04.079986 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:04.079990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:04.080048 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:04.080139 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:04.080514 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:04.081143 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:04.082271 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:04.082588 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:04.082693 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:04.083863 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:04.083866 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:04.083868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:25:04.083880 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:04.083906 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:25:04.083916 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:04.084017 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:04.084138 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 20:25:04.084190 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:25:04.085547 21918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3705e97eaaea4abba1008c4b9b55f475. Latest consensus state: current_term: 1 leader_uuid: "3705e97eaaea4abba1008c4b9b55f475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3705e97eaaea4abba1008c4b9b55f475" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36415 } } }
I20260926 20:25:04.085682 21918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475 [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:04.085971 21917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3705e97eaaea4abba1008c4b9b55f475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3705e97eaaea4abba1008c4b9b55f475" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36415 } } }
I20260926 20:25:04.086067 21917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475 [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:04.086102 21927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:25:04.089355 21927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:25:04.095436 21927 catalog_manager.cc:1347] Generated new cluster ID: 7f74064cd55c458ca148c01de416f8a8
I20260926 20:25:04.095551 21927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:25:04.119781 21927 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:25:04.121088 21927 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:25:04.126717 21927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3705e97eaaea4abba1008c4b9b55f475: Generated new TSK 0
I20260926 20:25:04.127636 21927 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:25:04.136037 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 21838
W20260926 20:25:04.141820 21823 connection.cc:535] client connection to 127.0.0.1:36415 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36415): Connection reset by peer (error 104)
2026-09-26T20:25:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:25:04 dist-test-slave-qsm9 krb5kdc[21937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:25:04 dist-test-slave-qsm9 krb5kdc[21937](info): set up 2 sockets
Sep 26 20:25:04 dist-test-slave-qsm9 krb5kdc[21937](info): commencing operation
krb5kdc: starting...
W20260926 20:25:05.192366 467 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:25:05 dist-test-slave-qsm9 krb5kdc[21937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:25:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:25:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:25:05.323134 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34915
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:25:05.438089 21950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:25:05.438439 21950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:25:05.438539 21950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:25:05.442719 21950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:25:05.442840 21950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:25:05.442883 21950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:25:05.442921 21950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:25:05.442977 21950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:25:05.448684 21950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34915
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:25:05.449890 21950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:25:05.450970 21950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:25:05.459848 21955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:05.459931 21958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:05.460016 21950 server_base.cc:1034] running on GCE node
W20260926 20:25:05.459852 21956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:05.460767 21950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:25:05.461906 21950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:25:05.463088 21950 hybrid_clock.cc:648] HybridClock initialized: now 1790454305463051 us; error 51 us; skew 500 ppm
Sep 26 20:25:05 dist-test-slave-qsm9 krb5kdc[21937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:25:05.466224 21950 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:25:05.467139 21950 webserver.cc:460] Webserver started at http://127.0.0.1:46609/ using document root <none> and password file <none>
I20260926 20:25:05.467775 21950 fs_manager.cc:359] Metadata directory not provided
I20260926 20:25:05.467840 21950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:25:05.468021 21950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:25:05.470110 21950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "c57aa63b9848412597e9f0f8d2ffddab"
format_stamp: "Formatted at 2026-09-26 20:25:05 on dist-test-slave-qsm9"
server_key: "c7209b58eb0f2f389b7335b634fea8c9"
server_key_iv: "7b9bf3c5518cf37c94f1c7ef1df47372"
server_key_version: "encryptionkey@0"
I20260926 20:25:05.470623 21950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "c57aa63b9848412597e9f0f8d2ffddab"
format_stamp: "Formatted at 2026-09-26 20:25:05 on dist-test-slave-qsm9"
server_key: "c7209b58eb0f2f389b7335b634fea8c9"
server_key_iv: "7b9bf3c5518cf37c94f1c7ef1df47372"
server_key_version: "encryptionkey@0"
I20260926 20:25:05.474471 21950 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 20:25:05.477159 21965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:25:05.478435 21950 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:25:05.478579 21950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "c57aa63b9848412597e9f0f8d2ffddab"
format_stamp: "Formatted at 2026-09-26 20:25:05 on dist-test-slave-qsm9"
server_key: "c7209b58eb0f2f389b7335b634fea8c9"
server_key_iv: "7b9bf3c5518cf37c94f1c7ef1df47372"
server_key_version: "encryptionkey@0"
I20260926 20:25:05.478693 21950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:25:05.483317 21950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:25:05.486363 21950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:25:05.486565 21950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:25:05.493566 21950 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34915
I20260926 20:25:05.493629 22017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34915 every 8 connection(s)
I20260926 20:25:05.494598 21950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:25:05.497678 22018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:25:05.500095 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 21950
I20260926 20:25:05.500264 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:25:05.500588 467 external_mini_cluster.cc:1428] Setting key ed0ab172c1250512b1591f9c1ed482e3
I20260926 20:25:05.504755 22018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab: Bootstrap starting.
I20260926 20:25:05.506804 22021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:05.502135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:59263 (local address 127.0.0.1:34915)
0926 20:25:05.502623 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:05.502644 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:05.502689 (+ 45us) server_negotiation.cc:407] Connection header received
0926 20:25:05.503577 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:05.503604 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:05.504053 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:05.504448 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:05.505335 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:05.505341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:05.505356 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:25:05.505368 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:05.505439 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:25:05.505468 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:05.505606 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:05.505681 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 20:25:05.505841 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":131,"threads_started":1}
I20260926 20:25:05.507671 22018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab: Neither blocks nor log segments found. Creating new log.
I20260926 20:25:05.508488 22018 log.cc:824] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab: Log is configured to *not* fsync() on all Append() calls
I20260926 20:25:05.509897 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:25:05.510706 22018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab: No bootstrap required, opened a new log
I20260926 20:25:05.513813 22021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:05.512072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.224:38215 (local address 127.0.0.1:34915)
0926 20:25:05.512215 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:05.512219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:05.512359 (+ 140us) server_negotiation.cc:407] Connection header received
0926 20:25:05.512511 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:05.512514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:05.512549 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:05.512641 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:05.513395 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:05.513399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:05.513402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:25:05.513414 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:05.513444 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:25:05.513466 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:05.513558 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:05.513638 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 20:25:05.513680 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:25:05.514945 22018 raft_consensus.cc:348] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57aa63b9848412597e9f0f8d2ffddab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34915 } }
I20260926 20:25:05.515198 22018 raft_consensus.cc:374] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:25:05.515260 22018 raft_consensus.cc:729] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57aa63b9848412597e9f0f8d2ffddab, State: Initialized, Role: FOLLOWER
I20260926 20:25:05.515769 22018 consensus_queue.cc:260] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57aa63b9848412597e9f0f8d2ffddab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34915 } }
I20260926 20:25:05.515892 22018 raft_consensus.cc:388] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:25:05.515981 22018 raft_consensus.cc:482] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:25:05.516095 22018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:25:05.517200 22018 raft_consensus.cc:504] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57aa63b9848412597e9f0f8d2ffddab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34915 } }
I20260926 20:25:05.517575 22018 leader_election.cc:302] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c57aa63b9848412597e9f0f8d2ffddab; no voters:
I20260926 20:25:05.517891 22018 leader_election.cc:288] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:25:05.518138 22028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:25:05.518392 22028 raft_consensus.cc:686] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab [term 1 LEADER]: Becoming Leader. State: Replica: c57aa63b9848412597e9f0f8d2ffddab, State: Running, Role: LEADER
I20260926 20:25:05.518703 22028 consensus_queue.cc:237] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57aa63b9848412597e9f0f8d2ffddab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34915 } }
I20260926 20:25:05.519266 22018 sys_catalog.cc:564] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:25:05.520471 22030 sys_catalog.cc:455] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab [sys.catalog]: SysCatalogTable state changed. Reason: New leader c57aa63b9848412597e9f0f8d2ffddab. Latest consensus state: current_term: 1 leader_uuid: "c57aa63b9848412597e9f0f8d2ffddab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57aa63b9848412597e9f0f8d2ffddab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34915 } } }
I20260926 20:25:05.520612 22030 sys_catalog.cc:458] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:05.521004 22037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:25:05.520958 22029 sys_catalog.cc:455] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c57aa63b9848412597e9f0f8d2ffddab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57aa63b9848412597e9f0f8d2ffddab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34915 } } }
I20260926 20:25:05.521114 22029 sys_catalog.cc:458] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:05.524449 22037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:25:05.529933 22037 catalog_manager.cc:1347] Generated new cluster ID: d2af70c43a654e6f985690fb400b9b7a
I20260926 20:25:05.530022 22037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:25:05.547318 22037 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:25:05.547786 22037 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:25:05.555902 22037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c57aa63b9848412597e9f0f8d2ffddab: Generated new TSK 0
I20260926 20:25:05.556799 22037 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:25:05.559962 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 21950
2026-09-26T20:25:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:25:05 dist-test-slave-qsm9 krb5kdc[22049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:25:05 dist-test-slave-qsm9 krb5kdc[22049](info): set up 2 sockets
Sep 26 20:25:05 dist-test-slave-qsm9 krb5kdc[22049](info): commencing operation
krb5kdc: starting...
W20260926 20:25:06.592151 467 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:25:06 dist-test-slave-qsm9 krb5kdc[22049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:25:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:25:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:25:06.724166 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:40893
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:43745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:40893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:25:06.840785 22062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:25:06.841146 22062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:25:06.841243 22062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:25:06.845405 22062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:25:06.845528 22062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:25:06.845574 22062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:25:06.845618 22062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:25:06.845654 22062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:25:06.851704 22062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:43745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:40893
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:40893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:25:06.853166 22062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:25:06.854301 22062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:25:06.863364 22067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:06.863463 22062 server_base.cc:1034] running on GCE node
W20260926 20:25:06.863374 22068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:06.863714 22070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:06.864348 22062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:25:06.865556 22062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:25:06.866827 22062 hybrid_clock.cc:648] HybridClock initialized: now 1790454306866785 us; error 50 us; skew 500 ppm
Sep 26 20:25:06 dist-test-slave-qsm9 krb5kdc[22049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:25:06.869946 22062 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:25:06.870951 22062 webserver.cc:460] Webserver started at http://127.0.116.254:32903/ using document root <none> and password file <none>
I20260926 20:25:06.871598 22062 fs_manager.cc:359] Metadata directory not provided
I20260926 20:25:06.871691 22062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:25:06.871925 22062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:25:06.874092 22062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "890f6d8ad0164ff3b9771e869c35ec82"
format_stamp: "Formatted at 2026-09-26 20:25:06 on dist-test-slave-qsm9"
server_key: "af447e482578008efd021f9d2a0d0961"
server_key_iv: "8b99607056dcc693de245098c972307a"
server_key_version: "encryptionkey@0"
I20260926 20:25:06.874764 22062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "890f6d8ad0164ff3b9771e869c35ec82"
format_stamp: "Formatted at 2026-09-26 20:25:06 on dist-test-slave-qsm9"
server_key: "af447e482578008efd021f9d2a0d0961"
server_key_iv: "8b99607056dcc693de245098c972307a"
server_key_version: "encryptionkey@0"
I20260926 20:25:06.879264 22062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:25:06.881933 22077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:25:06.883271 22062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:25:06.883452 22062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "890f6d8ad0164ff3b9771e869c35ec82"
format_stamp: "Formatted at 2026-09-26 20:25:06 on dist-test-slave-qsm9"
server_key: "af447e482578008efd021f9d2a0d0961"
server_key_iv: "8b99607056dcc693de245098c972307a"
server_key_version: "encryptionkey@0"
I20260926 20:25:06.883632 22062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:25:06.901909 22062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:25:06.905493 22062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:25:06.905716 22062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:25:06.913268 22062 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:40893
I20260926 20:25:06.913291 22129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:40893 every 8 connection(s)
I20260926 20:25:06.914317 22062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:25:06.917289 22130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:25:06.921176 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 22062
I20260926 20:25:06.921348 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:25:06.921644 467 external_mini_cluster.cc:1428] Setting key 856e54620f522aa4d72835b70027234b
I20260926 20:25:06.923417 22130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82: Bootstrap starting.
I20260926 20:25:06.926151 22130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82: Neither blocks nor log segments found. Creating new log.
I20260926 20:25:06.926890 22130 log.cc:824] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82: Log is configured to *not* fsync() on all Append() calls
I20260926 20:25:06.928768 22130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82: No bootstrap required, opened a new log
Sep 26 20:25:06 dist-test-slave-qsm9 krb5kdc[22049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:25:06.932478 22130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890f6d8ad0164ff3b9771e869c35ec82" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 40893 } }
I20260926 20:25:06.932777 22130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:25:06.932874 22130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890f6d8ad0164ff3b9771e869c35ec82, State: Initialized, Role: FOLLOWER
I20260926 20:25:06.933405 22130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890f6d8ad0164ff3b9771e869c35ec82" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 40893 } }
I20260926 20:25:06.933574 22130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:25:06.933667 22130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:25:06.933768 22130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:25:06.934653 22130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890f6d8ad0164ff3b9771e869c35ec82" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 40893 } }
I20260926 20:25:06.935000 22130 leader_election.cc:302] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 890f6d8ad0164ff3b9771e869c35ec82; no voters:
I20260926 20:25:06.935338 22130 leader_election.cc:288] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:25:06.935405 22135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:25:06.935627 22135 raft_consensus.cc:686] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82 [term 1 LEADER]: Becoming Leader. State: Replica: 890f6d8ad0164ff3b9771e869c35ec82, State: Running, Role: LEADER
I20260926 20:25:06.935981 22135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890f6d8ad0164ff3b9771e869c35ec82" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 40893 } }
I20260926 20:25:06.936650 22130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:25:06.937909 22136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 890f6d8ad0164ff3b9771e869c35ec82. Latest consensus state: current_term: 1 leader_uuid: "890f6d8ad0164ff3b9771e869c35ec82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890f6d8ad0164ff3b9771e869c35ec82" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 40893 } } }
I20260926 20:25:06.937959 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:06.923254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44765 (local address 127.0.116.254:40893)
0926 20:25:06.923797 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:06.923816 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:06.923857 (+ 41us) server_negotiation.cc:407] Connection header received
0926 20:25:06.924697 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:06.924725 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:06.925176 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:06.925538 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:06.926319 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:06.927617 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:06.928569 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:06.929136 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:06.929309 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:06.932131 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:06.932137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:06.932151 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:06.932179 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:06.934294 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:06.934855 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:06.934861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:06.934868 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:06.934994 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:06.935365 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:06.935369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:06.935373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:06.936050 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:06.936372 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:06.936708 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 20:25:06.936948 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":225,"threads_started":1}
I20260926 20:25:06.938048 22136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82 [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:06.938618 22137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "890f6d8ad0164ff3b9771e869c35ec82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890f6d8ad0164ff3b9771e869c35ec82" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 40893 } } }
I20260926 20:25:06.938766 22137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82 [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:06.941298 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 20:25:06.942054 22150 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:25:06.942129 22150 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:25:06.942188 22144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:25:06.943043 22144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:25:06.949205 22144 catalog_manager.cc:1347] Generated new cluster ID: 3cc2fa32f0864d8db32e6b3701f736dd
I20260926 20:25:06.949270 22144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:25:06.954735 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:06.945586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52237 (local address 127.0.116.254:40893)
0926 20:25:06.945715 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:06.945719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:06.945734 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:25:06.945772 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:06.945775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:06.945819 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:06.945906 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:06.946737 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:06.947743 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:06.948816 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:06.949183 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:06.949291 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:06.951180 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:06.951184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:06.951196 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:06.951221 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:06.952919 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:06.953455 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:06.953458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:06.953460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:06.953540 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:06.953876 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:06.953879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:06.953881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:06.954069 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:06.954200 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:06.954478 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 20:25:06.954571 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:25:06.964038 22144 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:25:06.965523 22144 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:25:06.973381 22144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 890f6d8ad0164ff3b9771e869c35ec82: Generated new TSK 0
I20260926 20:25:06.974112 22144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:25:06.981438 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 22062
W20260926 20:25:06.986840 22047 connection.cc:535] client connection to 127.0.116.254:40893 recv error: Network error: failed to read from TLS socket (remote: 127.0.116.254:40893): Connection reset by peer (error 104)
2026-09-26T20:25:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:25:07 dist-test-slave-qsm9 krb5kdc[22161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:25:07 dist-test-slave-qsm9 krb5kdc[22161](info): set up 2 sockets
Sep 26 20:25:07 dist-test-slave-qsm9 krb5kdc[22161](info): commencing operation
krb5kdc: starting...
W20260926 20:25:08.052451 467 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:25:08 dist-test-slave-qsm9 krb5kdc[22161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:25:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:25:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:25:08.178710 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:41681
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:33641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:41681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:25:08.290834 22174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:25:08.291133 22174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:25:08.291177 22174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:25:08.295301 22174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:25:08.295378 22174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:25:08.295398 22174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:25:08.295418 22174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:25:08.295432 22174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:25:08.301143 22174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:33641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:41681
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:41681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:25:08.302275 22174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:25:08.303292 22174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:25:08.312388 22180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:08.312434 22182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:08.312433 22179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:08.312634 22174 server_base.cc:1034] running on GCE node
I20260926 20:25:08.313328 22174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:25:08.314476 22174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:25:08.315652 22174 hybrid_clock.cc:648] HybridClock initialized: now 1790454308315624 us; error 42 us; skew 500 ppm
Sep 26 20:25:08 dist-test-slave-qsm9 krb5kdc[22161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:25:08.318424 22174 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:25:08.319329 22174 webserver.cc:460] Webserver started at http://127.0.116.254:43621/ using document root <none> and password file <none>
I20260926 20:25:08.319945 22174 fs_manager.cc:359] Metadata directory not provided
I20260926 20:25:08.320039 22174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:25:08.320278 22174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:25:08.322361 22174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "9aa40b9477af40e7b6b8dcdf2d0853af"
format_stamp: "Formatted at 2026-09-26 20:25:08 on dist-test-slave-qsm9"
server_key: "d6b4c795148bd5d7d32cc568df39f0c5"
server_key_iv: "521ebc023b79cc316b88e77eae0f2fe5"
server_key_version: "encryptionkey@0"
I20260926 20:25:08.322944 22174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "9aa40b9477af40e7b6b8dcdf2d0853af"
format_stamp: "Formatted at 2026-09-26 20:25:08 on dist-test-slave-qsm9"
server_key: "d6b4c795148bd5d7d32cc568df39f0c5"
server_key_iv: "521ebc023b79cc316b88e77eae0f2fe5"
server_key_version: "encryptionkey@0"
I20260926 20:25:08.326812 22174 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:25:08.329255 22189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:25:08.330489 22174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 20:25:08.330657 22174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "9aa40b9477af40e7b6b8dcdf2d0853af"
format_stamp: "Formatted at 2026-09-26 20:25:08 on dist-test-slave-qsm9"
server_key: "d6b4c795148bd5d7d32cc568df39f0c5"
server_key_iv: "521ebc023b79cc316b88e77eae0f2fe5"
server_key_version: "encryptionkey@0"
I20260926 20:25:08.330776 22174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:25:08.350778 22174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:25:08.354180 22174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:25:08.354350 22174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:25:08.361726 22174 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:41681
I20260926 20:25:08.361756 22241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:41681 every 8 connection(s)
I20260926 20:25:08.362777 22174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:25:08.365756 22242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:25:08.365955 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 22174
I20260926 20:25:08.366132 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:25:08.366420 467 external_mini_cluster.cc:1428] Setting key fc9eedbf3ea1fffdf906ef42f513daef
I20260926 20:25:08.373214 22242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af: Bootstrap starting.
I20260926 20:25:08.376226 22242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af: Neither blocks nor log segments found. Creating new log.
I20260926 20:25:08.377029 22245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:08.367959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39689 (local address 127.0.116.254:41681)
0926 20:25:08.368440 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:08.368460 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:08.368504 (+ 44us) server_negotiation.cc:407] Connection header received
0926 20:25:08.369303 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:08.369328 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:08.369790 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:08.370169 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:08.370587 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:08.372127 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:08.373025 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:08.373702 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:08.373861 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:08.374826 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:08.374833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:08.374850 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:25:08.375382 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:08.375463 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:25:08.375529 (+ 66us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:08.375660 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:08.375872 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 20:25:08.376014 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":122,"threads_started":1}
I20260926 20:25:08.377163 22242 log.cc:824] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af: Log is configured to *not* fsync() on all Append() calls
I20260926 20:25:08.379390 22242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af: No bootstrap required, opened a new log
I20260926 20:25:08.380559 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:25:08.383291 22242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa40b9477af40e7b6b8dcdf2d0853af" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41681 } }
I20260926 20:25:08.383595 22242 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:25:08.383664 22242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa40b9477af40e7b6b8dcdf2d0853af, State: Initialized, Role: FOLLOWER
I20260926 20:25:08.384169 22242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa40b9477af40e7b6b8dcdf2d0853af" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41681 } }
I20260926 20:25:08.384320 22242 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:25:08.384397 22242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:25:08.384522 22242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:25:08.385646 22242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa40b9477af40e7b6b8dcdf2d0853af" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41681 } }
I20260926 20:25:08.386041 22242 leader_election.cc:302] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9aa40b9477af40e7b6b8dcdf2d0853af; no voters:
I20260926 20:25:08.386420 22242 leader_election.cc:288] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:25:08.386543 22252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:25:08.386782 22252 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af [term 1 LEADER]: Becoming Leader. State: Replica: 9aa40b9477af40e7b6b8dcdf2d0853af, State: Running, Role: LEADER
I20260926 20:25:08.387331 22252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa40b9477af40e7b6b8dcdf2d0853af" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41681 } }
I20260926 20:25:08.387471 22245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:08.382651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47769 (local address 127.0.116.254:41681)
0926 20:25:08.382786 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:08.382790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:08.382989 (+ 199us) server_negotiation.cc:407] Connection header received
0926 20:25:08.383149 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:08.383153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:08.383204 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:08.383290 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:08.383606 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:08.384354 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:08.385256 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:08.385671 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:08.385780 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:08.386792 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:08.386796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:08.386799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:25:08.386813 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:08.386842 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:25:08.386855 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:08.386943 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:08.387272 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 20:25:08.387309 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:25:08.388032 22242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:25:08.389144 22253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9aa40b9477af40e7b6b8dcdf2d0853af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa40b9477af40e7b6b8dcdf2d0853af" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41681 } } }
I20260926 20:25:08.389271 22253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:08.389479 22254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9aa40b9477af40e7b6b8dcdf2d0853af. Latest consensus state: current_term: 1 leader_uuid: "9aa40b9477af40e7b6b8dcdf2d0853af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa40b9477af40e7b6b8dcdf2d0853af" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 41681 } } }
I20260926 20:25:08.389672 22254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:08.390543 22262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:25:08.393468 22262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:25:08.399372 22262 catalog_manager.cc:1347] Generated new cluster ID: 03ddfaceffa34a1dbac29b4def675fb1
I20260926 20:25:08.399458 22262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:25:08.413792 22262 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:25:08.415356 22262 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:25:08.420167 22262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9aa40b9477af40e7b6b8dcdf2d0853af: Generated new TSK 0
I20260926 20:25:08.420912 22262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:25:08.426024 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 22174
W20260926 20:25:08.430738 22159 connection.cc:535] client connection to 127.0.116.254:41681 recv error: Network error: failed to read from TLS socket (remote: 127.0.116.254:41681): Connection reset by peer (error 104)
2026-09-26T20:25:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:25:08 dist-test-slave-qsm9 krb5kdc[22273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:25:08 dist-test-slave-qsm9 krb5kdc[22273](info): set up 2 sockets
Sep 26 20:25:08 dist-test-slave-qsm9 krb5kdc[22273](info): commencing operation
krb5kdc: starting...
W20260926 20:25:09.480163 467 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:25:09 dist-test-slave-qsm9 krb5kdc[22273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:25:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:25:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:25:09.606703 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:39487
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:39487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:25:09.720779 22286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:25:09.721127 22286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:25:09.721223 22286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:25:09.725674 22286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:25:09.725783 22286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:25:09.725822 22286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:25:09.725883 22286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:25:09.725939 22286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:25:09.731683 22286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:39487
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:39487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:25:09.732867 22286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:25:09.733968 22286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:25:09.743012 22291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:09.743109 22286 server_base.cc:1034] running on GCE node
W20260926 20:25:09.743081 22292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:09.743288 22294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:09.744005 22286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:25:09.745226 22286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:25:09.746408 22286 hybrid_clock.cc:648] HybridClock initialized: now 1790454309746384 us; error 33 us; skew 500 ppm
Sep 26 20:25:09 dist-test-slave-qsm9 krb5kdc[22273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:25:09.753970 22286 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:25:09.755028 22286 webserver.cc:460] Webserver started at http://127.0.116.254:39987/ using document root <none> and password file <none>
I20260926 20:25:09.755692 22286 fs_manager.cc:359] Metadata directory not provided
I20260926 20:25:09.755771 22286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:25:09.755980 22286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:25:09.758159 22286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "d03978c79770406c9914d1be5d053f07"
format_stamp: "Formatted at 2026-09-26 20:25:09 on dist-test-slave-qsm9"
server_key: "d38fc82055cf522374b07f8c08d461bd"
server_key_iv: "bd832a0b473baec39b556529830f12f3"
server_key_version: "encryptionkey@0"
I20260926 20:25:09.758719 22286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "d03978c79770406c9914d1be5d053f07"
format_stamp: "Formatted at 2026-09-26 20:25:09 on dist-test-slave-qsm9"
server_key: "d38fc82055cf522374b07f8c08d461bd"
server_key_iv: "bd832a0b473baec39b556529830f12f3"
server_key_version: "encryptionkey@0"
I20260926 20:25:09.762749 22286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:25:09.765228 22301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:25:09.766484 22286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:25:09.766628 22286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "d03978c79770406c9914d1be5d053f07"
format_stamp: "Formatted at 2026-09-26 20:25:09 on dist-test-slave-qsm9"
server_key: "d38fc82055cf522374b07f8c08d461bd"
server_key_iv: "bd832a0b473baec39b556529830f12f3"
server_key_version: "encryptionkey@0"
I20260926 20:25:09.766738 22286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:25:09.771602 22286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:25:09.774924 22286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:25:09.775141 22286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:25:09.782109 22286 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:39487
I20260926 20:25:09.782128 22353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:39487 every 8 connection(s)
I20260926 20:25:09.783113 22286 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:25:09.785815 22354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:25:09.791181 22354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07: Bootstrap starting.
I20260926 20:25:09.793252 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 22286
I20260926 20:25:09.793419 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:25:09.793717 467 external_mini_cluster.cc:1428] Setting key f9a5e20a7fe578095e9a55a622fe4b97
I20260926 20:25:09.793730 22354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07: Neither blocks nor log segments found. Creating new log.
I20260926 20:25:09.794612 22354 log.cc:824] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07: Log is configured to *not* fsync() on all Append() calls
I20260926 20:25:09.796734 22354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07: No bootstrap required, opened a new log
I20260926 20:25:09.798715 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:09.794864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40995 (local address 127.0.116.254:39487)
0926 20:25:09.795333 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:09.795355 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:09.795402 (+ 47us) server_negotiation.cc:407] Connection header received
0926 20:25:09.796140 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:09.796158 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:09.796429 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:09.796709 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:09.797528 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:09.797532 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:09.797544 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:25:09.797553 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:09.797603 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:25:09.797621 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:09.797706 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:09.797846 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 20:25:09.798037 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":150,"threads_started":1}
I20260926 20:25:09.800665 22354 raft_consensus.cc:348] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03978c79770406c9914d1be5d053f07" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 39487 } }
I20260926 20:25:09.800974 22354 raft_consensus.cc:374] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:25:09.801085 22354 raft_consensus.cc:729] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d03978c79770406c9914d1be5d053f07, State: Initialized, Role: FOLLOWER
I20260926 20:25:09.801468 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:25:09.801578 22354 consensus_queue.cc:260] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03978c79770406c9914d1be5d053f07" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 39487 } }
I20260926 20:25:09.801743 22354 raft_consensus.cc:388] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:25:09.801807 22354 raft_consensus.cc:482] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:25:09.801901 22354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:25:09.802870 22354 raft_consensus.cc:504] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03978c79770406c9914d1be5d053f07" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 39487 } }
I20260926 20:25:09.803211 22354 leader_election.cc:302] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d03978c79770406c9914d1be5d053f07; no voters:
I20260926 20:25:09.803581 22354 leader_election.cc:288] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:25:09.803654 22363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:25:09.803880 22363 raft_consensus.cc:686] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07 [term 1 LEADER]: Becoming Leader. State: Replica: d03978c79770406c9914d1be5d053f07, State: Running, Role: LEADER
I20260926 20:25:09.804258 22363 consensus_queue.cc:237] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03978c79770406c9914d1be5d053f07" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 39487 } }
I20260926 20:25:09.804916 22354 sys_catalog.cc:564] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:25:09.805841 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:09.803576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51879 (local address 127.0.116.254:39487)
0926 20:25:09.803827 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:09.803833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:09.804068 (+ 235us) server_negotiation.cc:407] Connection header received
0926 20:25:09.804201 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:09.804204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:09.804239 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:09.804318 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:09.805126 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:09.805130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:09.805133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:25:09.805145 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:09.805174 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:25:09.805186 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:09.805277 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:09.805624 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 20:25:09.805667 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260926 20:25:09.805899 22366 sys_catalog.cc:455] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d03978c79770406c9914d1be5d053f07. Latest consensus state: current_term: 1 leader_uuid: "d03978c79770406c9914d1be5d053f07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03978c79770406c9914d1be5d053f07" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 39487 } } }
I20260926 20:25:09.806017 22366 sys_catalog.cc:458] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07 [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:09.806368 22365 sys_catalog.cc:455] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d03978c79770406c9914d1be5d053f07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d03978c79770406c9914d1be5d053f07" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 39487 } } }
I20260926 20:25:09.806529 22365 sys_catalog.cc:458] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07 [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:09.807235 22373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:25:09.809772 22373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:25:09.814829 22373 catalog_manager.cc:1347] Generated new cluster ID: 0c6e85ab6ca2462d8a71ad6cd9e772cb
I20260926 20:25:09.814909 22373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:25:09.828233 22373 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:25:09.828558 22373 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:25:09.834554 22373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d03978c79770406c9914d1be5d053f07: Generated new TSK 0
I20260926 20:25:09.835361 22373 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:25:09.846951 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 22286
2026-09-26T20:25:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1421 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:25:09 dist-test-slave-qsm9 krb5kdc[22385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:25:09 dist-test-slave-qsm9 krb5kdc[22385](info): set up 2 sockets
Sep 26 20:25:09 dist-test-slave-qsm9 krb5kdc[22385](info): commencing operation
krb5kdc: starting...
W20260926 20:25:10.880173 467 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:25:10 dist-test-slave-qsm9 krb5kdc[22385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:25:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:25:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:25:11.007345 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:42791
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:42791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:25:11.121766 22398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:25:11.122148 22398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:25:11.122246 22398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:25:11.126730 22398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:25:11.126832 22398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:25:11.126874 22398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:25:11.126940 22398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:25:11.126983 22398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:25:11.133253 22398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:42791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:42791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:25:11.134459 22398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:25:11.135602 22398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:25:11.144420 22404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:11.144425 22406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:11.144420 22403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:11.144750 22398 server_base.cc:1034] running on GCE node
I20260926 20:25:11.145349 22398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:25:11.146595 22398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:25:11.147815 22398 hybrid_clock.cc:648] HybridClock initialized: now 1790454311147777 us; error 45 us; skew 500 ppm
Sep 26 20:25:11 dist-test-slave-qsm9 krb5kdc[22385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:25:11.150848 22398 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:25:11.151961 22398 webserver.cc:460] Webserver started at http://127.0.116.254:36811/ using document root <none> and password file <none>
I20260926 20:25:11.152526 22398 fs_manager.cc:359] Metadata directory not provided
I20260926 20:25:11.152587 22398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:25:11.152756 22398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:25:11.155169 22398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "25f8b70776024579bedf3b88338b6db5"
format_stamp: "Formatted at 2026-09-26 20:25:11 on dist-test-slave-qsm9"
server_key: "b39271f8cad27ff14416a8801d30b0b7"
server_key_iv: "2cf33e4a3d68ac1952e1272f0fc12e32"
server_key_version: "encryptionkey@0"
I20260926 20:25:11.155867 22398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "25f8b70776024579bedf3b88338b6db5"
format_stamp: "Formatted at 2026-09-26 20:25:11 on dist-test-slave-qsm9"
server_key: "b39271f8cad27ff14416a8801d30b0b7"
server_key_iv: "2cf33e4a3d68ac1952e1272f0fc12e32"
server_key_version: "encryptionkey@0"
I20260926 20:25:11.160184 22398 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:25:11.162678 22413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:25:11.164208 22398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:25:11.164356 22398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "25f8b70776024579bedf3b88338b6db5"
format_stamp: "Formatted at 2026-09-26 20:25:11 on dist-test-slave-qsm9"
server_key: "b39271f8cad27ff14416a8801d30b0b7"
server_key_iv: "2cf33e4a3d68ac1952e1272f0fc12e32"
server_key_version: "encryptionkey@0"
I20260926 20:25:11.164480 22398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:25:11.188360 22398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:25:11.192003 22398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:25:11.192236 22398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:25:11.199759 22398 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:42791
I20260926 20:25:11.199776 22465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:42791 every 8 connection(s)
I20260926 20:25:11.200791 22398 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:25:11.203722 22466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:25:11.204154 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 22398
I20260926 20:25:11.204299 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:25:11.204603 467 external_mini_cluster.cc:1428] Setting key 99b85bd2e0f855db6e3c82aa371a9a9d
I20260926 20:25:11.210608 22466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5: Bootstrap starting.
I20260926 20:25:11.213027 22466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5: Neither blocks nor log segments found. Creating new log.
Sep 26 20:25:11 dist-test-slave-qsm9 krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:25:11.213843 22466 log.cc:824] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:25:11.216112 22466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5: No bootstrap required, opened a new log
I20260926 20:25:11.219014 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:11.206053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59485 (local address 127.0.116.254:42791)
0926 20:25:11.206477 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:11.206492 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:11.206527 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:25:11.207248 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:11.207266 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:11.207651 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:11.208052 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:11.208865 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:11.209847 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:11.210800 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:11.211290 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:11.211471 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:11.214133 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:11.214141 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:11.214160 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:11.214190 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:11.216136 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:11.216626 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:11.216631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:11.216636 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:11.216747 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:11.217057 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:11.217060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:11.217062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:11.217463 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:11.217675 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:11.217964 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 20:25:11.218272 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":121,"threads_started":1}
I20260926 20:25:11.219908 22466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f8b70776024579bedf3b88338b6db5" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 42791 } }
I20260926 20:25:11.220135 22466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:25:11.220194 22466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25f8b70776024579bedf3b88338b6db5, State: Initialized, Role: FOLLOWER
I20260926 20:25:11.220641 22466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f8b70776024579bedf3b88338b6db5" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 42791 } }
I20260926 20:25:11.220790 22466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:25:11.220883 22466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:25:11.221010 22466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:25:11.222008 22466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f8b70776024579bedf3b88338b6db5" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 42791 } }
I20260926 20:25:11.222363 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:25:11.222410 22466 leader_election.cc:302] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25f8b70776024579bedf3b88338b6db5; no voters:
I20260926 20:25:11.222743 22466 leader_election.cc:288] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:25:11.222918 22471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:25:11.223224 22471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5 [term 1 LEADER]: Becoming Leader. State: Replica: 25f8b70776024579bedf3b88338b6db5, State: Running, Role: LEADER
I20260926 20:25:11.223577 22471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f8b70776024579bedf3b88338b6db5" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 42791 } }
I20260926 20:25:11.224182 22466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:25:11.225431 22472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25f8b70776024579bedf3b88338b6db5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f8b70776024579bedf3b88338b6db5" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 42791 } } }
I20260926 20:25:11.225569 22472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5 [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:11.225973 22473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25f8b70776024579bedf3b88338b6db5. Latest consensus state: current_term: 1 leader_uuid: "25f8b70776024579bedf3b88338b6db5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f8b70776024579bedf3b88338b6db5" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 42791 } } }
I20260926 20:25:11.226075 22473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5 [sys.catalog]: This master's current role is: LEADER
W20260926 20:25:11.229559 22491 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:25:11.229691 22491 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:25:11.229756 22486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:25:11.230597 22486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:25:11.234532 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:11.224689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36223 (local address 127.0.116.254:42791)
0926 20:25:11.224819 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:11.224823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:11.226700 (+ 1877us) server_negotiation.cc:407] Connection header received
0926 20:25:11.226912 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:11.226915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:11.226961 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:11.227057 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:11.227822 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:11.228357 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:11.229201 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:11.229462 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:11.229568 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:11.231073 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:11.231077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:11.231080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:11.231112 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:11.232690 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:11.233171 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:11.233174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:11.233176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:11.233261 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:11.233571 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:11.233574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:11.233576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:11.233792 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:11.233889 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:11.234240 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 20:25:11.234371 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:25:11.236903 22486 catalog_manager.cc:1347] Generated new cluster ID: 18c98623b5f54699bf6166c69fe4b234
I20260926 20:25:11.237025 22486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:25:11.249782 22486 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:25:11.250816 22486 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:25:11.258037 22486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25f8b70776024579bedf3b88338b6db5: Generated new TSK 0
I20260926 20:25:11.258819 22486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:25:11.270051 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 22398
2026-09-26T20:25:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:25:11 dist-test-slave-qsm9 krb5kdc[22497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:25:11 dist-test-slave-qsm9 krb5kdc[22497](info): set up 2 sockets
Sep 26 20:25:11 dist-test-slave-qsm9 krb5kdc[22497](info): commencing operation
krb5kdc: starting...
W20260926 20:25:12.304132 467 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:25:12 dist-test-slave-qsm9 krb5kdc[22497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:25:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:25:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:25:12.434828 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:45915
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:45915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:25:12.546793 22510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:25:12.547082 22510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:25:12.547127 22510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:25:12.551122 22510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:25:12.551188 22510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:25:12.551209 22510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:25:12.551225 22510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:25:12.551241 22510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:25:12.556569 22510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:45915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:45915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:25:12.557637 22510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:25:12.558629 22510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:25:12.567451 22515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:12.567440 22516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:12.567543 22510 server_base.cc:1034] running on GCE node
W20260926 20:25:12.567456 22518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:12.568316 22510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:25:12.569425 22510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:25:12.570618 22510 hybrid_clock.cc:648] HybridClock initialized: now 1790454312570590 us; error 40 us; skew 500 ppm
Sep 26 20:25:12 dist-test-slave-qsm9 krb5kdc[22497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454312, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:25:12.573480 22510 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:25:12.574561 22510 webserver.cc:460] Webserver started at http://127.0.116.254:33375/ using document root <none> and password file <none>
I20260926 20:25:12.575230 22510 fs_manager.cc:359] Metadata directory not provided
I20260926 20:25:12.575330 22510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:25:12.575636 22510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:25:12.577910 22510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "fd8b02f0073d4546899a18f5be3a5e3a"
format_stamp: "Formatted at 2026-09-26 20:25:12 on dist-test-slave-qsm9"
server_key: "02b1b049ccb04d0f168ccb57e9e0469e"
server_key_iv: "e269b732a985aff897195bf94e6add92"
server_key_version: "encryptionkey@0"
I20260926 20:25:12.578528 22510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "fd8b02f0073d4546899a18f5be3a5e3a"
format_stamp: "Formatted at 2026-09-26 20:25:12 on dist-test-slave-qsm9"
server_key: "02b1b049ccb04d0f168ccb57e9e0469e"
server_key_iv: "e269b732a985aff897195bf94e6add92"
server_key_version: "encryptionkey@0"
I20260926 20:25:12.582788 22510 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:25:12.585731 22525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:25:12.587108 22510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 20:25:12.587261 22510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "fd8b02f0073d4546899a18f5be3a5e3a"
format_stamp: "Formatted at 2026-09-26 20:25:12 on dist-test-slave-qsm9"
server_key: "02b1b049ccb04d0f168ccb57e9e0469e"
server_key_iv: "e269b732a985aff897195bf94e6add92"
server_key_version: "encryptionkey@0"
I20260926 20:25:12.587396 22510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:25:12.619716 22510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:25:12.623108 22510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:25:12.623338 22510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:25:12.630999 22510 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:45915
I20260926 20:25:12.631039 22577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:45915 every 8 connection(s)
I20260926 20:25:12.632157 22510 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:25:12.635046 22578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:25:12.640456 22578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a: Bootstrap starting.
I20260926 20:25:12.641834 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 22510
I20260926 20:25:12.641996 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:25:12.642256 467 external_mini_cluster.cc:1428] Setting key 289b9a63e69a67253ca6e17dc3ca6cb4
I20260926 20:25:12.643136 22578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a: Neither blocks nor log segments found. Creating new log.
I20260926 20:25:12.644109 22578 log.cc:824] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:25:12.646239 22578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a: No bootstrap required, opened a new log
I20260926 20:25:12.650720 22578 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8b02f0073d4546899a18f5be3a5e3a" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45915 } }
I20260926 20:25:12.651011 22578 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:25:12.651105 22578 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8b02f0073d4546899a18f5be3a5e3a, State: Initialized, Role: FOLLOWER
I20260926 20:25:12.651610 22578 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8b02f0073d4546899a18f5be3a5e3a" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45915 } }
I20260926 20:25:12.651777 22578 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:25:12.651841 22578 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:25:12.651937 22578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:25:12.652302 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:12.643436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52249 (local address 127.0.116.254:45915)
0926 20:25:12.643952 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:12.643979 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:12.644035 (+ 56us) server_negotiation.cc:407] Connection header received
0926 20:25:12.644805 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:12.644833 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:12.645252 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:12.645609 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:12.646020 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:12.647461 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:12.648433 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:12.649228 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:12.649439 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:12.650208 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:12.650215 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:12.650231 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:25:12.650720 (+ 489us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:12.650792 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:25:12.650822 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:12.650920 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:12.651031 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 20:25:12.651253 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":149,"threads_started":1}
I20260926 20:25:12.652992 22578 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8b02f0073d4546899a18f5be3a5e3a" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45915 } }
I20260926 20:25:12.653385 22578 leader_election.cc:302] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd8b02f0073d4546899a18f5be3a5e3a; no voters:
I20260926 20:25:12.653731 22578 leader_election.cc:288] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:25:12.653918 22583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:25:12.654573 22583 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a [term 1 LEADER]: Becoming Leader. State: Replica: fd8b02f0073d4546899a18f5be3a5e3a, State: Running, Role: LEADER
I20260926 20:25:12.655025 22578 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:25:12.654989 22583 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8b02f0073d4546899a18f5be3a5e3a" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45915 } }
I20260926 20:25:12.655541 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:25:12.656850 22585 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd8b02f0073d4546899a18f5be3a5e3a. Latest consensus state: current_term: 1 leader_uuid: "fd8b02f0073d4546899a18f5be3a5e3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8b02f0073d4546899a18f5be3a5e3a" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45915 } } }
I20260926 20:25:12.656999 22585 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:12.657380 22584 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd8b02f0073d4546899a18f5be3a5e3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8b02f0073d4546899a18f5be3a5e3a" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45915 } } }
I20260926 20:25:12.657480 22584 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:12.657579 22601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:25:12.660579 22601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:25:12.665550 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:12.660252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37339 (local address 127.0.116.254:45915)
0926 20:25:12.660362 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:12.660366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:12.660659 (+ 293us) server_negotiation.cc:407] Connection header received
0926 20:25:12.660838 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:12.660842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:12.660891 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:12.660976 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:12.661297 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:12.662077 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:12.663270 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:12.663641 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:12.663747 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:12.665113 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:12.665118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:12.665121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:25:12.665141 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:12.665167 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:25:12.665177 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:12.665266 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:12.665360 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 20:25:12.665411 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:25:12.666675 22601 catalog_manager.cc:1347] Generated new cluster ID: 41ec3e664648445f9e1192d205e0c64d
I20260926 20:25:12.666752 22601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:25:12.676363 22601 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:25:12.677670 22601 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:25:12.695286 22601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd8b02f0073d4546899a18f5be3a5e3a: Generated new TSK 0
I20260926 20:25:12.696213 22601 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:25:12.707216 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 22510
2026-09-26T20:25:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:25:12 dist-test-slave-qsm9 krb5kdc[22609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:25:12 dist-test-slave-qsm9 krb5kdc[22609](info): set up 2 sockets
Sep 26 20:25:12 dist-test-slave-qsm9 krb5kdc[22609](info): commencing operation
krb5kdc: starting...
W20260926 20:25:13.752259 467 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:25:13 dist-test-slave-qsm9 krb5kdc[22609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:25:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:25:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:25:13.879781 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:45751
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:45751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:25:13.994385 22622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:25:13.994724 22622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:25:13.994812 22622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:25:13.999076 22622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:25:13.999202 22622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:25:13.999239 22622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:25:13.999277 22622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:25:13.999316 22622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:25:14.005102 22622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:45751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:45751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:25:14.006304 22622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:25:14.007406 22622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:25:14.016268 22628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:14.016256 22630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:14.016270 22627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:14.016598 22622 server_base.cc:1034] running on GCE node
I20260926 20:25:14.017285 22622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:25:14.018508 22622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:25:14.019680 22622 hybrid_clock.cc:648] HybridClock initialized: now 1790454314019647 us; error 45 us; skew 500 ppm
Sep 26 20:25:14 dist-test-slave-qsm9 krb5kdc[22609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:25:14.022773 22622 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:25:14.023797 22622 webserver.cc:460] Webserver started at http://127.0.116.254:43145/ using document root <none> and password file <none>
I20260926 20:25:14.024354 22622 fs_manager.cc:359] Metadata directory not provided
I20260926 20:25:14.024462 22622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:25:14.024701 22622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:25:14.026743 22622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "29319f82f32c4f9faf12d541b7cf595e"
format_stamp: "Formatted at 2026-09-26 20:25:14 on dist-test-slave-qsm9"
server_key: "34bc2d22263728d35f766b2978ce3601"
server_key_iv: "c9d79d9e52a5559c661ce59797d6941e"
server_key_version: "encryptionkey@0"
I20260926 20:25:14.027311 22622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "29319f82f32c4f9faf12d541b7cf595e"
format_stamp: "Formatted at 2026-09-26 20:25:14 on dist-test-slave-qsm9"
server_key: "34bc2d22263728d35f766b2978ce3601"
server_key_iv: "c9d79d9e52a5559c661ce59797d6941e"
server_key_version: "encryptionkey@0"
I20260926 20:25:14.031070 22622 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:25:14.033753 22637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:25:14.035032 22622 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:25:14.035174 22622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "29319f82f32c4f9faf12d541b7cf595e"
format_stamp: "Formatted at 2026-09-26 20:25:14 on dist-test-slave-qsm9"
server_key: "34bc2d22263728d35f766b2978ce3601"
server_key_iv: "c9d79d9e52a5559c661ce59797d6941e"
server_key_version: "encryptionkey@0"
I20260926 20:25:14.035295 22622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:25:14.040103 22622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:25:14.043385 22622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:25:14.043630 22622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:25:14.050889 22622 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:45751
I20260926 20:25:14.050906 22689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:45751 every 8 connection(s)
I20260926 20:25:14.051923 22622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:25:14.054653 22690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:25:14.056267 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 22622
I20260926 20:25:14.056407 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:25:14.056696 467 external_mini_cluster.cc:1428] Setting key 1e9607080c1d02f9755c410352e41c2b
I20260926 20:25:14.061003 22690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e: Bootstrap starting.
I20260926 20:25:14.062103 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:14.058109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56909 (local address 127.0.116.254:45751)
0926 20:25:14.058573 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:14.058591 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:14.058640 (+ 49us) server_negotiation.cc:407] Connection header received
0926 20:25:14.059305 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:14.059322 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:14.059684 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:14.060058 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:14.060906 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:14.060911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:14.060922 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:25:14.060933 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:14.060989 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:25:14.061011 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:14.061169 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:14.061243 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 20:25:14.061414 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":128,"threads_started":1}
I20260926 20:25:14.063583 22690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e: Neither blocks nor log segments found. Creating new log.
I20260926 20:25:14.064441 22690 log.cc:824] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:25:14.065258 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:25:14.066509 22690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e: No bootstrap required, opened a new log
I20260926 20:25:14.068989 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:14.067285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38157 (local address 127.0.116.254:45751)
0926 20:25:14.067398 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:14.067401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:14.067676 (+ 275us) server_negotiation.cc:407] Connection header received
0926 20:25:14.067792 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:14.067794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:14.067823 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:14.067890 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:14.068583 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:14.068586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:14.068588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:25:14.068598 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:14.068622 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:25:14.068632 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:14.068699 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:14.068810 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 20:25:14.068843 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:25:14.070799 22690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29319f82f32c4f9faf12d541b7cf595e" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45751 } }
I20260926 20:25:14.071040 22690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:25:14.071098 22690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29319f82f32c4f9faf12d541b7cf595e, State: Initialized, Role: FOLLOWER
I20260926 20:25:14.071599 22690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29319f82f32c4f9faf12d541b7cf595e" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45751 } }
I20260926 20:25:14.071748 22690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:25:14.071815 22690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:25:14.071908 22690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:25:14.072906 22690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29319f82f32c4f9faf12d541b7cf595e" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45751 } }
I20260926 20:25:14.073244 22690 leader_election.cc:302] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29319f82f32c4f9faf12d541b7cf595e; no voters:
I20260926 20:25:14.073565 22690 leader_election.cc:288] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:25:14.073724 22700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:25:14.074060 22700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e [term 1 LEADER]: Becoming Leader. State: Replica: 29319f82f32c4f9faf12d541b7cf595e, State: Running, Role: LEADER
I20260926 20:25:14.074410 22700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29319f82f32c4f9faf12d541b7cf595e" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45751 } }
I20260926 20:25:14.074945 22690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:25:14.076172 22702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29319f82f32c4f9faf12d541b7cf595e. Latest consensus state: current_term: 1 leader_uuid: "29319f82f32c4f9faf12d541b7cf595e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29319f82f32c4f9faf12d541b7cf595e" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45751 } } }
I20260926 20:25:14.076306 22702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:14.076699 22709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:25:14.076642 22701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29319f82f32c4f9faf12d541b7cf595e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29319f82f32c4f9faf12d541b7cf595e" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 45751 } } }
I20260926 20:25:14.076830 22701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:14.079996 22709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:25:14.085741 22709 catalog_manager.cc:1347] Generated new cluster ID: e288a77359c74462bd897073a1fbdfcc
I20260926 20:25:14.085815 22709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:25:14.115960 22709 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:25:14.116364 22709 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:25:14.124681 22709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29319f82f32c4f9faf12d541b7cf595e: Generated new TSK 0
I20260926 20:25:14.125449 22709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:25:14.135206 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 22622
2026-09-26T20:25:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1426 ms)
[----------] 18 tests from AuthTokenIssuingTest (25968 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:25:14 dist-test-slave-qsm9 krb5kdc[22721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:25:14 dist-test-slave-qsm9 krb5kdc[22721](info): set up 2 sockets
Sep 26 20:25:14 dist-test-slave-qsm9 krb5kdc[22721](info): commencing operation
krb5kdc: starting...
W20260926 20:25:15.176247 467 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:25:15 dist-test-slave-qsm9 krb5kdc[22721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:25:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:25:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:25:15.304716 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:36523
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:36523,127.0.116.253:40121,127.0.116.252:36867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:25:15.422559 22734 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:25:15.422951 22734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:25:15.423020 22734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:25:15.423103 22734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:25:15.427304 22734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:25:15.427412 22734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:25:15.427452 22734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:25:15.427527 22734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:25:15.427595 22734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:25:15.433272 22734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:36523,127.0.116.253:40121,127.0.116.252:36867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:36523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:25:15.434466 22734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:25:15.435561 22734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:25:15.444600 22742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:15.444845 22734 server_base.cc:1034] running on GCE node
W20260926 20:25:15.444600 22740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:15.444797 22739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:15.445693 22734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:25:15.446907 22734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:25:15.448092 22734 hybrid_clock.cc:648] HybridClock initialized: now 1790454315448077 us; error 29 us; skew 500 ppm
Sep 26 20:25:15 dist-test-slave-qsm9 krb5kdc[22721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:25:15.455981 22734 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:25:15.457155 22734 webserver.cc:460] Webserver started at http://127.0.116.254:43269/ using document root <none> and password file <none>
I20260926 20:25:15.457770 22734 fs_manager.cc:359] Metadata directory not provided
I20260926 20:25:15.457866 22734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:25:15.458122 22734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:25:15.460408 22734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "5400d8c430994f5397171a981e6392ca"
format_stamp: "Formatted at 2026-09-26 20:25:15 on dist-test-slave-qsm9"
server_key: "cdc23a970cd3eb3e43cfdd2675aba09e"
server_key_iv: "37e7f8c819e3b09c65ee4282378c0139"
server_key_version: "encryptionkey@0"
I20260926 20:25:15.461023 22734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "5400d8c430994f5397171a981e6392ca"
format_stamp: "Formatted at 2026-09-26 20:25:15 on dist-test-slave-qsm9"
server_key: "cdc23a970cd3eb3e43cfdd2675aba09e"
server_key_iv: "37e7f8c819e3b09c65ee4282378c0139"
server_key_version: "encryptionkey@0"
I20260926 20:25:15.465098 22734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:25:15.467795 22749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:25:15.469127 22734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:25:15.469290 22734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "5400d8c430994f5397171a981e6392ca"
format_stamp: "Formatted at 2026-09-26 20:25:15 on dist-test-slave-qsm9"
server_key: "cdc23a970cd3eb3e43cfdd2675aba09e"
server_key_iv: "37e7f8c819e3b09c65ee4282378c0139"
server_key_version: "encryptionkey@0"
I20260926 20:25:15.469424 22734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:25:15.479274 22734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:25:15.482530 22734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:25:15.482751 22734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:25:15.490319 22734 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:36523
I20260926 20:25:15.490613 22801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:36523 every 8 connection(s)
I20260926 20:25:15.491421 22734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:25:15.494467 22802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:25:15.498512 22802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36523 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:25:15.501287 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 22734
I20260926 20:25:15.501453 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:25:15.501755 467 external_mini_cluster.cc:1428] Setting key e7e810bd26f9c11469e5f70c5f818ab4
WARNING: no policy specified for kudu/127.0.116.253@KRBTEST.COM; defaulting to no policy
Sep 26 20:25:15 dist-test-slave-qsm9 krb5kdc[22721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.253@KRBTEST.COM" created.
I20260926 20:25:15.517086 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.501043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41974 (local address 127.0.116.254:36523)
0926 20:25:15.501623 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:15.501640 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:15.501945 (+ 305us) server_negotiation.cc:407] Connection header received
0926 20:25:15.503694 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:15.503703 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:15.503994 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:15.504206 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:15.505075 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.506525 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.507776 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.508467 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.508641 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:15.511245 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:15.511251 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:15.511263 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:15.511286 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:15.513339 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:15.513950 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:15.513960 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:15.513964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:15.514111 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:15.514522 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:15.514529 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:15.514533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:15.515121 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:15.515285 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:15.515883 (+ 598us) server_negotiation.cc:299] Negotiation successful
0926 20:25:15.516166 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":454,"spinlock_wait_cycles":77184,"thread_start_us":154,"threads_started":1}
I20260926 20:25:15.517091 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.501043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:36523 (local address 127.0.0.1:41974)
0926 20:25:15.501701 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.501769 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 20:25:15.503245 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:25:15.504223 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:25:15.504228 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:25:15.504622 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:25:15.504897 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:15.504916 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.506678 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.506689 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:15.507647 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:15.507659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.508642 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.508646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:15.508749 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:15.509426 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:25:15.509445 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:25:15.511073 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:25:15.513496 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:15.513510 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:15.513526 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:15.513811 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:15.514271 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:15.514273 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:15.514275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:15.514415 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:15.515299 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:25:15.515308 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:25:15.515546 (+ 238us) client_negotiation.cc:769] Sending connection context
0926 20:25:15.515788 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 20:25:15.516291 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":113,"thread_start_us":154,"threads_started":1}
I20260926 20:25:15.519743 22802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:25:15.520674 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.520408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:40121 (local address 127.0.0.1:55402)
0926 20:25:15.520535 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.520609 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.253:40121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 20:25:15.520967 22753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.116.253:40121: connect: Connection refused (error 111)
I20260926 20:25:15.521834 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.521639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:40121 (local address 127.0.0.1:55406)
0926 20:25:15.521741 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.521784 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.253:40121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 20:25:15.522089 22802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.253:40121: Network error: Client connection negotiation failed: client connection to 127.0.116.253:40121: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.253.keytab.
Entry for principal kudu/127.0.116.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.253.keytab.
I20260926 20:25:15.531476 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.253:40121
--webserver_interface=127.0.116.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.253.keytab
--principal=kudu/127.0.116.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:36523,127.0.116.253:40121,127.0.116.252:36867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:25:15.571357 22802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } attempt: 1
I20260926 20:25:15.572312 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.572004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:40121 (local address 127.0.0.1:55422)
0926 20:25:15.572171 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.572235 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.253:40121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 20:25:15.573089 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.572868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:40121 (local address 127.0.0.1:55426)
0926 20:25:15.572994 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.573037 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.253:40121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260926 20:25:15.573235 22802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.253:40121: Network error: Client connection negotiation failed: client connection to 127.0.116.253:40121: connect: Connection refused (error 111)
I20260926 20:25:15.641440 22802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } attempt: 2
I20260926 20:25:15.642371 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.642021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:40121 (local address 127.0.0.1:55440)
0926 20:25:15.642208 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.642289 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.253:40121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 20:25:15.643146 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.642948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:40121 (local address 127.0.0.1:55442)
0926 20:25:15.643049 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.643095 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.253:40121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 20:25:15.643303 22802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.253:40121: Network error: Client connection negotiation failed: client connection to 127.0.116.253:40121: connect: Connection refused (error 111)
W20260926 20:25:15.648164 22809 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:25:15.648420 22809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:25:15.648443 22809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:25:15.648540 22809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:25:15.652791 22809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:25:15.652842 22809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:25:15.652863 22809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:25:15.652878 22809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:25:15.652931 22809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:25:15.658463 22809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:36523,127.0.116.253:40121,127.0.116.252:36867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.253.keytab
--principal=kudu/127.0.116.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.253:40121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.116.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:25:15.659607 22809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:25:15.660665 22809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:25:15.669378 22817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:15.669365 22814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:15.669441 22815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:15.669463 22809 server_base.cc:1034] running on GCE node
I20260926 20:25:15.670291 22809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:25:15.671432 22809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:25:15.672621 22809 hybrid_clock.cc:648] HybridClock initialized: now 1790454315672570 us; error 62 us; skew 500 ppm
Sep 26 20:25:15 dist-test-slave-qsm9 krb5kdc[22721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:25:15.675611 22809 init.cc:377] Logged in from keytab as kudu/127.0.116.253@KRBTEST.COM (short username kudu)
I20260926 20:25:15.676512 22809 webserver.cc:460] Webserver started at http://127.0.116.253:42913/ using document root <none> and password file <none>
I20260926 20:25:15.677112 22809 fs_manager.cc:359] Metadata directory not provided
I20260926 20:25:15.677245 22809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:25:15.677476 22809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:25:15.679616 22809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-1/data/instance:
uuid: "1cdadadc4cf14d8abaed6f09dcebc343"
format_stamp: "Formatted at 2026-09-26 20:25:15 on dist-test-slave-qsm9"
server_key: "f0d02ec27c01061986bbdcfa1ae143a1"
server_key_iv: "b51fb5a4449c5030917e9d4c0cb9949a"
server_key_version: "encryptionkey@0"
I20260926 20:25:15.680190 22809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-1/wal/instance:
uuid: "1cdadadc4cf14d8abaed6f09dcebc343"
format_stamp: "Formatted at 2026-09-26 20:25:15 on dist-test-slave-qsm9"
server_key: "f0d02ec27c01061986bbdcfa1ae143a1"
server_key_iv: "b51fb5a4449c5030917e9d4c0cb9949a"
server_key_version: "encryptionkey@0"
I20260926 20:25:15.684032 22809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:25:15.686569 22824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:25:15.687934 22809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:25:15.688076 22809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-1/wal
uuid: "1cdadadc4cf14d8abaed6f09dcebc343"
format_stamp: "Formatted at 2026-09-26 20:25:15 on dist-test-slave-qsm9"
server_key: "f0d02ec27c01061986bbdcfa1ae143a1"
server_key_iv: "b51fb5a4449c5030917e9d4c0cb9949a"
server_key_version: "encryptionkey@0"
I20260926 20:25:15.688192 22809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:25:15.721237 22809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:25:15.724574 22809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:25:15.724795 22809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:25:15.732432 22809 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.253:40121
I20260926 20:25:15.732812 22876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.253:40121 every 8 connection(s)
I20260926 20:25:15.733516 22809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-1/data/info.pb
I20260926 20:25:15.734467 22802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } attempt: 3
I20260926 20:25:15.737111 22877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:25:15.738456 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 22809
I20260926 20:25:15.738574 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-1/wal/instance
I20260926 20:25:15.738912 467 external_mini_cluster.cc:1428] Setting key dafa04e8562b2c33ac91f6d030cb698b
I20260926 20:25:15.741511 22877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36523 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:25:15 dist-test-slave-qsm9 krb5kdc[22721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for kudu/127.0.116.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.116.252@KRBTEST.COM; defaulting to no policy
I20260926 20:25:15.750001 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.735028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:40121 (local address 127.0.0.1:55446)
0926 20:25:15.735202 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.735221 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:25:15.735335 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:25:15.738150 (+ 2815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:25:15.738153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:25:15.738400 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:25:15.738563 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:15.738572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.740511 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.740515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:15.741301 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:15.741315 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.742404 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.742409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:15.742522 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:15.743152 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:25:15.743170 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:25:15.744925 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:25:15.748087 (+ 3162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:15.748094 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:15.748099 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:15.748353 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:15.748805 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:15.748808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:15.748809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:15.748897 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:15.749577 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:25:15.749579 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:25:15.749668 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 20:25:15.749753 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 20:25:15.749868 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.252@KRBTEST.COM" created.
Sep 26 20:25:15 dist-test-slave-qsm9 krb5kdc[22721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.253@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:25:15.751308 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.736128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55446 (local address 127.0.116.253:40121)
0926 20:25:15.736665 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:15.736686 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:15.736734 (+ 48us) server_negotiation.cc:407] Connection header received
0926 20:25:15.737467 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:15.737486 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:15.737832 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:15.738135 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:15.738731 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.740357 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.741446 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.742247 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.742432 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:15.745131 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:15.745138 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:15.745150 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:15.745182 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:15.747945 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:15.748490 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:15.748496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:15.748512 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:15.748685 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:15.748994 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:15.748999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:15.749001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:15.749465 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:15.749565 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:15.749872 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 20:25:15.750291 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":165,"threads_started":1}
I20260926 20:25:15.754426 22802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:25:15.755263 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.754960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:36867 (local address 127.0.0.1:47528)
0926 20:25:15.755098 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.755160 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:36867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 20:25:15.756119 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.743397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:36523 (local address 127.0.0.1:41988)
0926 20:25:15.744050 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.744102 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:25:15.744267 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:25:15.744659 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:25:15.744665 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:25:15.745047 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:25:15.745214 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:15.745223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.746326 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.746330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:15.746958 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:15.746966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.747709 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.747713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:15.747862 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:15.748467 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:25:15.748482 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:25:15.751789 (+ 3307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:25:15.753873 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:15.753885 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:15.753893 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:15.754245 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:15.754631 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:15.754637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:15.754642 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:15.754738 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:15.755260 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:25:15.755270 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:25:15.755422 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 20:25:15.755596 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 20:25:15.755831 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"spinlock_wait_cycles":12032,"thread_start_us":133,"threads_started":1}
I20260926 20:25:15.756992 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.743453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41988 (local address 127.0.116.254:36523)
0926 20:25:15.743643 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:15.743650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:15.744174 (+ 524us) server_negotiation.cc:407] Connection header received
0926 20:25:15.744381 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:15.744385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:15.744442 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:15.744528 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:15.745351 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.746209 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.747083 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.747575 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.747695 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:15.751946 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:15.751949 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:15.751951 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:15.751974 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:15.753724 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:15.754420 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:15.754426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:15.754431 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:15.754524 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:15.754849 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:15.754855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:15.754859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:15.755130 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:15.756523 (+ 1393us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:15.756620 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 20:25:15.756769 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 20:25:15.757678 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.755959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:36867 (local address 127.0.0.1:47544)
0926 20:25:15.757560 (+ 1601us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.757613 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:36867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1533}
W20260926 20:25:15.757956 22802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.252:36867: Network error: Client connection negotiation failed: client connection to 127.0.116.252:36867: connect: Connection refused (error 111)
I20260926 20:25:15.758380 22877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:25:15 dist-test-slave-qsm9 krb5kdc[22721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.253@KRBTEST.COM for kudu/127.0.116.253@KRBTEST.COM
I20260926 20:25:15.766976 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.758886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:40121 (local address 127.0.0.1:55454)
0926 20:25:15.758990 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.759002 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:25:15.759079 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:25:15.759333 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:25:15.759335 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:25:15.759528 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:25:15.759639 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:15.759644 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.760684 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.760687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:15.761292 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:15.761301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.761716 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.761720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:15.761822 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:15.762322 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:25:15.762334 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:25:15.763682 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:25:15.765529 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:15.765535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:15.765538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:15.765722 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:15.765995 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:15.765998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:15.766000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:15.766080 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:15.766443 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:25:15.766446 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:25:15.766540 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 20:25:15.766645 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 20:25:15.766792 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 20:25:15.766976 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.759011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55454 (local address 127.0.116.253:40121)
0926 20:25:15.759118 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:15.759123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:15.759135 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:25:15.759169 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:15.759174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:15.759223 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:15.759317 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:15.759750 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.760580 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.761391 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.761618 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.761696 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:15.763820 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:15.763824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:15.763825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:15.763844 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:15.765413 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:15.765816 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:15.765823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:15.765827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:15.765900 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:15.766170 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:15.766173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:15.766175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:15.766344 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:15.766443 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:15.766686 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 20:25:15.766850 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":31}
I20260926 20:25:15.768529 22877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:25:15.769330 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.769071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:36867 (local address 127.0.0.1:47548)
0926 20:25:15.769184 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.769263 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:36867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.252.keytab.
Entry for principal kudu/127.0.116.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.252.keytab.
W20260926 20:25:15.769594 22825 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.116.252:36867: connect: Connection refused (error 111)
I20260926 20:25:15.770480 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.770293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:36867 (local address 127.0.0.1:47554)
0926 20:25:15.770388 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.770429 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:36867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 20:25:15.770679 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.252:36867
--webserver_interface=127.0.116.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.252.keytab
--principal=kudu/127.0.116.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:36523,127.0.116.253:40121,127.0.116.252:36867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:25:15.770676 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.252:36867: Network error: Client connection negotiation failed: client connection to 127.0.116.252:36867: connect: Connection refused (error 111)
I20260926 20:25:15.789150 22802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } attempt: 1
I20260926 20:25:15.790063 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.789715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:36867 (local address 127.0.0.1:47568)
0926 20:25:15.789890 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.789969 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:36867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 20:25:15.790853 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.790593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:36867 (local address 127.0.0.1:47576)
0926 20:25:15.790711 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.790764 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:36867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 20:25:15.791015 22802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.252:36867: Network error: Client connection negotiation failed: client connection to 127.0.116.252:36867: connect: Connection refused (error 111)
I20260926 20:25:15.819980 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } attempt: 1
I20260926 20:25:15.820864 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.820560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:36867 (local address 127.0.0.1:47578)
0926 20:25:15.820713 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.820783 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:36867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 20:25:15.821682 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.821430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:36867 (local address 127.0.0.1:47594)
0926 20:25:15.821565 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.821618 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:36867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 20:25:15.821867 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.252:36867: Network error: Client connection negotiation failed: client connection to 127.0.116.252:36867: connect: Connection refused (error 111)
I20260926 20:25:15.866190 22802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } attempt: 2
I20260926 20:25:15.867079 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.866738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:36867 (local address 127.0.0.1:47600)
0926 20:25:15.866909 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.866986 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:36867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 20:25:15.867865 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.867636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:36867 (local address 127.0.0.1:47606)
0926 20:25:15.867744 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.867796 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:36867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 20:25:15.867995 22802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.252:36867: Network error: Client connection negotiation failed: client connection to 127.0.116.252:36867: connect: Connection refused (error 111)
W20260926 20:25:15.884085 22884 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:25:15.884378 22884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:25:15.884403 22884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:25:15.884442 22884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:25:15.888466 22884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:25:15.888522 22884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:25:15.888541 22884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:25:15.888558 22884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:25:15.888573 22884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:25:15.890035 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } attempt: 2
I20260926 20:25:15.890879 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.890596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:36867 (local address 127.0.0.1:47610)
0926 20:25:15.890743 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.890806 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:36867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 20:25:15.891635 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.891383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:36867 (local address 127.0.0.1:47612)
0926 20:25:15.891527 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.891582 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:36867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 20:25:15.891801 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.252:36867: Network error: Client connection negotiation failed: client connection to 127.0.116.252:36867: connect: Connection refused (error 111)
I20260926 20:25:15.894358 22884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:36523,127.0.116.253:40121,127.0.116.252:36867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/krb5kdc/kudu_127.0.116.252.keytab
--principal=kudu/127.0.116.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.252:36867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.116.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:25:15.895484 22884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:25:15.896591 22884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:25:15.905155 22890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:15.905256 22889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:15.905313 22884 server_base.cc:1034] running on GCE node
W20260926 20:25:15.905256 22892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:15.906378 22884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:25:15.907439 22884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:25:15.908661 22884 hybrid_clock.cc:648] HybridClock initialized: now 1790454315908633 us; error 44 us; skew 500 ppm
Sep 26 20:25:15 dist-test-slave-qsm9 krb5kdc[22721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:25:15.911562 22884 init.cc:377] Logged in from keytab as kudu/127.0.116.252@KRBTEST.COM (short username kudu)
I20260926 20:25:15.912483 22884 webserver.cc:460] Webserver started at http://127.0.116.252:32929/ using document root <none> and password file <none>
I20260926 20:25:15.913093 22884 fs_manager.cc:359] Metadata directory not provided
I20260926 20:25:15.913190 22884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:25:15.913427 22884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:25:15.915613 22884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-2/data/instance:
uuid: "3b27cc4c6099436fb7e6b08c16d79ac9"
format_stamp: "Formatted at 2026-09-26 20:25:15 on dist-test-slave-qsm9"
server_key: "41a0e1aa6c4696b9ad06d43e3921bab8"
server_key_iv: "d46b67201d4367c51b26bc68790e5b2a"
server_key_version: "encryptionkey@0"
I20260926 20:25:15.916258 22884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-2/wal/instance:
uuid: "3b27cc4c6099436fb7e6b08c16d79ac9"
format_stamp: "Formatted at 2026-09-26 20:25:15 on dist-test-slave-qsm9"
server_key: "41a0e1aa6c4696b9ad06d43e3921bab8"
server_key_iv: "d46b67201d4367c51b26bc68790e5b2a"
server_key_version: "encryptionkey@0"
I20260926 20:25:15.920126 22884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:25:15.922640 22899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:25:15.924042 22884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:25:15.924187 22884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-2/wal
uuid: "3b27cc4c6099436fb7e6b08c16d79ac9"
format_stamp: "Formatted at 2026-09-26 20:25:15 on dist-test-slave-qsm9"
server_key: "41a0e1aa6c4696b9ad06d43e3921bab8"
server_key_iv: "d46b67201d4367c51b26bc68790e5b2a"
server_key_version: "encryptionkey@0"
I20260926 20:25:15.924305 22884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:25:15.935976 22884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:25:15.939257 22884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:25:15.939464 22884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:25:15.946182 22884 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.252:36867
I20260926 20:25:15.946487 22951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.252:36867 every 8 connection(s)
I20260926 20:25:15.947204 22884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-2/data/info.pb
I20260926 20:25:15.950086 22952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:25:15.953842 22952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36523 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:25:15.956984 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 22884
I20260926 20:25:15.957122 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790454033974695-467-0/minicluster-data/master-2/wal/instance
I20260926 20:25:15.957389 467 external_mini_cluster.cc:1428] Setting key 6b8acb80466cbc93872cfe14130b9092
Sep 26 20:25:15 dist-test-slave-qsm9 krb5kdc[22721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.252@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
Sep 26 20:25:15 dist-test-slave-qsm9 krb5kdc[22721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.252@KRBTEST.COM
I20260926 20:25:15.967156 22802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } attempt: 3
I20260926 20:25:15.969331 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.955837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41996 (local address 127.0.116.254:36523)
0926 20:25:15.955992 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:15.955996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:15.956900 (+ 904us) server_negotiation.cc:407] Connection header received
0926 20:25:15.958021 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:15.958027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:15.958092 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:15.958196 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:15.959746 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.960581 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.961846 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.962146 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.962252 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:15.965020 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:15.965026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:15.965030 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:15.965066 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:15.966686 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:15.967528 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:15.967544 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:15.967546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:15.967642 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:15.968053 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:15.968057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:15.968058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:15.968320 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:15.968394 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:15.969055 (+ 661us) server_negotiation.cc:299] Negotiation successful
0926 20:25:15.969169 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 20:25:15.970343 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.958104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47616 (local address 127.0.116.252:36867)
0926 20:25:15.958447 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:15.958456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:15.958573 (+ 117us) server_negotiation.cc:407] Connection header received
0926 20:25:15.958697 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:15.958701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:15.958853 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:15.959011 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:15.959937 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.961032 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.962123 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.962570 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.962648 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:15.965528 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:15.965536 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:15.965551 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:15.965582 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:15.967552 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:15.968059 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:15.968065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:15.968066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:15.968148 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:15.968465 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:15.968468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:15.968470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:15.968776 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:15.968976 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:15.969608 (+ 632us) server_negotiation.cc:299] Negotiation successful
0926 20:25:15.969745 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":248,"thread_start_us":144,"threads_started":1}
I20260926 20:25:15.970343 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.956243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:36523 (local address 127.0.0.1:41996)
0926 20:25:15.956751 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.956794 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:25:15.957779 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:25:15.958404 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:25:15.958415 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:25:15.958872 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:25:15.959594 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:15.959615 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.960695 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.960698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:15.961721 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:15.961732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.962296 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.962299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:15.962457 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:15.963099 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:25:15.963122 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:25:15.964861 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:25:15.966806 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:15.966815 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:15.966832 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:15.967334 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:15.967750 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:15.967756 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:15.967760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:15.967941 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:15.968472 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:25:15.968478 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:25:15.968781 (+ 303us) client_negotiation.cc:769] Sending connection context
0926 20:25:15.968978 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 20:25:15.969306 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":32,"spinlock_wait_cycles":2816,"thread_start_us":141,"threads_started":1}
I20260926 20:25:15.971773 22952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:25:15 dist-test-slave-qsm9 krb5kdc[22721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for kudu/127.0.116.252@KRBTEST.COM
I20260926 20:25:15.977185 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.967673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:36867 (local address 127.0.0.1:47630)
0926 20:25:15.967865 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.967880 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:25:15.968004 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:25:15.968476 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:25:15.968479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:25:15.968642 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:25:15.968715 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:15.968720 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.969979 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.969984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:15.970591 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:15.970604 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.971555 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.971559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:15.971640 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:15.972110 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:25:15.972125 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:25:15.973591 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:25:15.975457 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:15.975460 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:15.975463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:15.975722 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:15.976014 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:15.976020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:15.976025 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:15.976132 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:15.976624 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:25:15.976630 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:25:15.976736 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 20:25:15.976854 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 20:25:15.976980 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260926 20:25:15.977211 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.967707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47630 (local address 127.0.116.252:36867)
0926 20:25:15.968275 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:15.968283 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:15.968301 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:25:15.968330 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:15.968332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:15.968364 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:15.968459 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:15.968823 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.969875 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.970953 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.971260 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.971414 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:15.973712 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:15.973717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:15.973722 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:15.973749 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:15.975351 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:15.975832 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:15.975836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:15.975837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:15.975914 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:15.976341 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:15.976344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:15.976345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:15.976534 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:15.976623 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:15.976920 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 20:25:15.977045 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":98,"threads_started":1}
Sep 26 20:25:15 dist-test-slave-qsm9 krb5kdc[22721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.252@KRBTEST.COM for kudu/127.0.116.253@KRBTEST.COM
Sep 26 20:25:15 dist-test-slave-qsm9 krb5kdc[22721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:25:15.981599 22802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca: Bootstrap starting.
I20260926 20:25:15.981863 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.972285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:40121 (local address 127.0.0.1:55468)
0926 20:25:15.972431 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.972445 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:25:15.972526 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:25:15.972861 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:25:15.972864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:25:15.973046 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:25:15.973636 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:15.973647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.974645 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.974648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:15.975555 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:15.975563 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.976024 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.976027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:15.976123 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:15.976573 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:25:15.976594 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:25:15.977988 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:25:15.980285 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:15.980288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:15.980290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:15.980532 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:15.980852 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:15.980854 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:15.980856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:15.980939 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:15.981398 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:25:15.981400 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:25:15.981485 (+ 85us) client_negotiation.cc:769] Sending connection context
0926 20:25:15.981589 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 20:25:15.981721 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 20:25:15.981997 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.972378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55468 (local address 127.0.116.253:40121)
0926 20:25:15.972501 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:15.972505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:15.972519 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:25:15.972619 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:15.972624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:15.972674 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:15.972744 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:15.973806 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.974513 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.975681 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.975908 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.976018 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:15.978129 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:15.978132 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:15.978135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:15.978163 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:15.980160 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:15.980650 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:15.980654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:15.980655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:15.980739 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:15.981051 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:15.981054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:15.981056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:15.981254 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:15.981325 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:15.981683 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 20:25:15.981804 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:25:15.982686 22952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:25:15.982997 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } attempt: 3
I20260926 20:25:15.984534 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.973420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42004 (local address 127.0.116.254:36523)
0926 20:25:15.973537 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:15.973543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:15.973557 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:25:15.973670 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:15.973673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:15.973714 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:15.973786 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:15.975237 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.975963 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.977008 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.977329 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.977403 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:15.980145 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:15.980148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:15.980151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:15.980173 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:15.981839 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:15.982794 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:15.982798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:15.982800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:15.982890 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:15.983617 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:15.983623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:15.983626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:15.983815 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:15.984022 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:15.984218 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 20:25:15.984336 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:25:15.984560 22802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca: Neither blocks nor log segments found. Creating new log.
I20260926 20:25:15.985813 22802 log.cc:824] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca: Log is configured to *not* fsync() on all Append() calls
I20260926 20:25:15.988114 22802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca: No bootstrap required, opened a new log
Sep 26 20:25:15 dist-test-slave-qsm9 krb5kdc[22721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.252@KRBTEST.COM for kudu/127.0.116.252@KRBTEST.COM
Sep 26 20:25:15 dist-test-slave-qsm9 krb5kdc[22721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.253@KRBTEST.COM for kudu/127.0.116.252@KRBTEST.COM
I20260926 20:25:15.993518 22802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5400d8c430994f5397171a981e6392ca" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36523 } } peers { permanent_uuid: "1cdadadc4cf14d8abaed6f09dcebc343" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } } peers { permanent_uuid: "3b27cc4c6099436fb7e6b08c16d79ac9" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } }
I20260926 20:25:15.993793 22802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5400d8c430994f5397171a981e6392ca, State: Initialized, Role: FOLLOWER
I20260926 20:25:15.994045 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.983333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:36867 (local address 127.0.0.1:47656)
0926 20:25:15.983581 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.983594 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:25:15.983664 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:25:15.984626 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:25:15.984628 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:25:15.984777 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:25:15.984846 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:15.984851 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.986029 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.986035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:15.986789 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:15.986798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.987481 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.987484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:15.987652 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:15.988127 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:25:15.988144 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:25:15.990502 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:25:15.992702 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:15.992705 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:15.992707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:15.992937 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:15.993170 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:15.993172 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:15.993174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:15.993237 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:15.993597 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:25:15.993599 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:25:15.993663 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 20:25:15.993743 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 20:25:15.993866 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":24}
I20260926 20:25:15.994153 22802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5400d8c430994f5397171a981e6392ca" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36523 } } peers { permanent_uuid: "1cdadadc4cf14d8abaed6f09dcebc343" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } } peers { permanent_uuid: "3b27cc4c6099436fb7e6b08c16d79ac9" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } }
I20260926 20:25:15.994443 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.983426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47656 (local address 127.0.116.252:36867)
0926 20:25:15.984133 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:15.984138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:15.984150 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:25:15.984181 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:15.984183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:15.984227 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:15.984298 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:15.985209 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.985918 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.986929 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.987294 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.987385 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:15.990613 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:15.990616 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:15.990618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:15.990659 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:15.992532 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:15.993026 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:15.993029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:15.993031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:15.993095 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:15.993311 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:15.993313 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:15.993315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:15.993490 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:15.993568 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:15.994226 (+ 658us) server_negotiation.cc:299] Negotiation successful
0926 20:25:15.994325 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620}
I20260926 20:25:15.994607 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.983366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:36867 (local address 127.0.0.1:47642)
0926 20:25:15.983478 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:15.983516 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 20:25:15.983588 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:25:15.985036 (+ 1448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:25:15.985039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:25:15.985212 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:25:15.985302 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:15.985308 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.986346 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.986352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:15.987143 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:15.987154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.987841 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.987845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:15.987925 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:15.988287 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:25:15.988300 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:25:15.990105 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:25:15.992850 (+ 2745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:15.992854 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:15.992856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:15.993111 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:15.993414 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:15.993417 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:15.993419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:15.993507 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:15.994109 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:25:15.994112 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:25:15.994216 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 20:25:15.994300 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:25:15.994440 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 20:25:15.995827 22802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:25:15.998507 22952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b27cc4c6099436fb7e6b08c16d79ac9: Bootstrap starting.
I20260926 20:25:16.001323 22952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b27cc4c6099436fb7e6b08c16d79ac9: Neither blocks nor log segments found. Creating new log.
I20260926 20:25:16.002439 22959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5400d8c430994f5397171a981e6392ca" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36523 } } peers { permanent_uuid: "1cdadadc4cf14d8abaed6f09dcebc343" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } } peers { permanent_uuid: "3b27cc4c6099436fb7e6b08c16d79ac9" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } } }
I20260926 20:25:16.002571 22959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 20:25:16 dist-test-slave-qsm9 krb5kdc[22721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.253@KRBTEST.COM
I20260926 20:25:16.005791 22952 log.cc:824] T 00000000000000000000000000000000 P 3b27cc4c6099436fb7e6b08c16d79ac9: Log is configured to *not* fsync() on all Append() calls
W20260926 20:25:16.006421 22970 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:25:16.006502 22970 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:25:16.006868 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.983557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47642 (local address 127.0.116.252:36867)
0926 20:25:15.984433 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:15.984436 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:15.984447 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:25:15.984477 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:15.984481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:15.984520 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:15.984997 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:15.985433 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.986213 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.987293 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.987708 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.988126 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:15.990254 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:15.990258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:15.990260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:15.990313 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:15.992723 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:15.993232 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:15.993235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:15.993237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:15.993316 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:15.993623 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:15.993626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:15.993628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:15.993981 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:16.006386 (+ 12405us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:16.006483 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 20:25:16.006741 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":802,"spinlock_wait_cycles":2304}
I20260926 20:25:16.004710 22734 master_runner.cc:186] Error getting master registration for 127.0.116.253:40121: OK, instance_id { permanent_uuid: "1cdadadc4cf14d8abaed6f09dcebc343" instance_seqno: 1790454315730227 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:25:16.008090 22734 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:25:16.010208 22877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cdadadc4cf14d8abaed6f09dcebc343: Bootstrap starting.
I20260926 20:25:16.012461 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:15.986091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.0.116.253:40121)
0926 20:25:15.986180 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:15.986184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:15.986328 (+ 144us) server_negotiation.cc:407] Connection header received
0926 20:25:15.986493 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:15.986496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:15.986534 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:15.986577 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:15.987383 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:15.988633 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:15.990509 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:16.001622 (+ 11113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:16.001721 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:16.008008 (+ 6287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:16.008012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:16.008016 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:16.008080 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:16.010119 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:16.010746 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:16.010750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:16.010752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:16.010845 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:16.011267 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:16.011270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:16.011272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:16.011632 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:16.011806 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:16.012176 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 20:25:16.012311 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 20:25:16.013199 22877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cdadadc4cf14d8abaed6f09dcebc343: Neither blocks nor log segments found. Creating new log.
I20260926 20:25:16.014158 22877 log.cc:824] T 00000000000000000000000000000000 P 1cdadadc4cf14d8abaed6f09dcebc343: Log is configured to *not* fsync() on all Append() calls
I20260926 20:25:16.016134 22952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b27cc4c6099436fb7e6b08c16d79ac9: No bootstrap required, opened a new log
I20260926 20:25:16.016285 22877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cdadadc4cf14d8abaed6f09dcebc343: No bootstrap required, opened a new log
I20260926 20:25:16.016999 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:25:16.018337 22776 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5400d8c430994f5397171a981e6392ca"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42004
I20260926 20:25:16.018597 22776 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 20:25:16.018716 22776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:25:16.020061 22776 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5400d8c430994f5397171a981e6392ca" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36523 } } peers { permanent_uuid: "1cdadadc4cf14d8abaed6f09dcebc343" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } } peers { permanent_uuid: "3b27cc4c6099436fb7e6b08c16d79ac9" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } }
I20260926 20:25:16.020459 22952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b27cc4c6099436fb7e6b08c16d79ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5400d8c430994f5397171a981e6392ca" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36523 } } peers { permanent_uuid: "1cdadadc4cf14d8abaed6f09dcebc343" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } } peers { permanent_uuid: "3b27cc4c6099436fb7e6b08c16d79ac9" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } }
I20260926 20:25:16.020533 22877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cdadadc4cf14d8abaed6f09dcebc343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5400d8c430994f5397171a981e6392ca" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36523 } } peers { permanent_uuid: "1cdadadc4cf14d8abaed6f09dcebc343" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } } peers { permanent_uuid: "3b27cc4c6099436fb7e6b08c16d79ac9" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } }
I20260926 20:25:16.020684 22952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b27cc4c6099436fb7e6b08c16d79ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b27cc4c6099436fb7e6b08c16d79ac9, State: Initialized, Role: FOLLOWER
I20260926 20:25:16.020730 22877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cdadadc4cf14d8abaed6f09dcebc343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cdadadc4cf14d8abaed6f09dcebc343, State: Initialized, Role: FOLLOWER
I20260926 20:25:16.021047 22776 leader_election.cc:288] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca [CANDIDATE]: Term 1 election: Requested vote from peers 1cdadadc4cf14d8abaed6f09dcebc343 (127.0.116.253:40121), 3b27cc4c6099436fb7e6b08c16d79ac9 (127.0.116.252:36867)
I20260926 20:25:16.020998 22952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b27cc4c6099436fb7e6b08c16d79ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5400d8c430994f5397171a981e6392ca" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36523 } } peers { permanent_uuid: "1cdadadc4cf14d8abaed6f09dcebc343" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } } peers { permanent_uuid: "3b27cc4c6099436fb7e6b08c16d79ac9" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } }
I20260926 20:25:16.021559 22877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cdadadc4cf14d8abaed6f09dcebc343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5400d8c430994f5397171a981e6392ca" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36523 } } peers { permanent_uuid: "1cdadadc4cf14d8abaed6f09dcebc343" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } } peers { permanent_uuid: "3b27cc4c6099436fb7e6b08c16d79ac9" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } }
I20260926 20:25:16.023334 22877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cdadadc4cf14d8abaed6f09dcebc343 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:25:16.023337 22952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b27cc4c6099436fb7e6b08c16d79ac9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:25:16.023371 22851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5400d8c430994f5397171a981e6392ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1cdadadc4cf14d8abaed6f09dcebc343"
I20260926 20:25:16.023568 22926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5400d8c430994f5397171a981e6392ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3b27cc4c6099436fb7e6b08c16d79ac9"
I20260926 20:25:16.023764 22926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b27cc4c6099436fb7e6b08c16d79ac9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:25:16.025671 22926 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3b27cc4c6099436fb7e6b08c16d79ac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5400d8c430994f5397171a981e6392ca in term 1.
I20260926 20:25:16.025978 22851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cdadadc4cf14d8abaed6f09dcebc343 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:25:16.026121 22750 leader_election.cc:302] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b27cc4c6099436fb7e6b08c16d79ac9, 5400d8c430994f5397171a981e6392ca; no voters:
I20260926 20:25:16.026432 22959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:25:16.026666 22959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca [term 1 LEADER]: Becoming Leader. State: Replica: 5400d8c430994f5397171a981e6392ca, State: Running, Role: LEADER
I20260926 20:25:16.026995 22959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5400d8c430994f5397171a981e6392ca" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36523 } } peers { permanent_uuid: "1cdadadc4cf14d8abaed6f09dcebc343" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } } peers { permanent_uuid: "3b27cc4c6099436fb7e6b08c16d79ac9" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } }
I20260926 20:25:16.027312 22851 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1cdadadc4cf14d8abaed6f09dcebc343 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5400d8c430994f5397171a981e6392ca in term 1.
W20260926 20:25:16.030069 23001 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3b27cc4c6099436fb7e6b08c16d79ac9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:25:16.030150 23001 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:25:16.030747 22959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5400d8c430994f5397171a981e6392ca. Latest consensus state: current_term: 1 leader_uuid: "5400d8c430994f5397171a981e6392ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5400d8c430994f5397171a981e6392ca" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36523 } } peers { permanent_uuid: "1cdadadc4cf14d8abaed6f09dcebc343" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } } peers { permanent_uuid: "3b27cc4c6099436fb7e6b08c16d79ac9" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } } }
I20260926 20:25:16.030889 22959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:16.031769 22974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b27cc4c6099436fb7e6b08c16d79ac9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5400d8c430994f5397171a981e6392ca" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36523 } } peers { permanent_uuid: "1cdadadc4cf14d8abaed6f09dcebc343" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } } peers { permanent_uuid: "3b27cc4c6099436fb7e6b08c16d79ac9" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } } }
I20260926 20:25:16.031893 22974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b27cc4c6099436fb7e6b08c16d79ac9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:25:16.032105 22975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cdadadc4cf14d8abaed6f09dcebc343 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5400d8c430994f5397171a981e6392ca" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36523 } } peers { permanent_uuid: "1cdadadc4cf14d8abaed6f09dcebc343" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } } peers { permanent_uuid: "3b27cc4c6099436fb7e6b08c16d79ac9" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } } }
I20260926 20:25:16.032227 22975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cdadadc4cf14d8abaed6f09dcebc343 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:25:16.034197 22809 master_runner.cc:263] No leader master found from master 1cdadadc4cf14d8abaed6f09dcebc343
I20260926 20:25:16.034265 22809 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 20:25:16.045456 23000 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1cdadadc4cf14d8abaed6f09dcebc343: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 20:25:16.045523 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:16.032653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47660 (local address 127.0.116.252:36867)
0926 20:25:16.032763 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:16.032768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:16.035246 (+ 2478us) server_negotiation.cc:407] Connection header received
0926 20:25:16.035438 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:16.035441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:16.035514 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:16.035573 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:16.036636 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:16.037430 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:16.038644 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:16.038991 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:16.039094 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:16.041101 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:16.041105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:16.041108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:16.041133 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:16.043583 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:16.044078 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:16.044081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:16.044083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:16.044168 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:16.044477 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:16.044480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:16.044482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:16.044726 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:16.044873 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:16.045229 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 20:25:16.045353 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 20:25:16.045544 23000 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:25:16.047788 23005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:25:16.048712 23005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:25:16.053740 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:16.033426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42014 (local address 127.0.116.254:36523)
0926 20:25:16.033529 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:16.033534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:16.038927 (+ 5393us) server_negotiation.cc:407] Connection header received
0926 20:25:16.039345 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:16.039349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:16.039403 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:16.039455 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:16.041723 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:16.042497 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:16.045606 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:16.045945 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:16.046035 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:16.049973 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:16.049977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:16.049980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:16.050023 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:16.051816 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:16.052308 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:16.052311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:16.052313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:16.052416 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:16.052709 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:16.052712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:16.052714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:16.052953 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:16.053069 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:16.053431 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 20:25:16.053569 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:25:16.054764 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:16.034142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55498 (local address 127.0.116.253:40121)
0926 20:25:16.034249 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:16.034253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:16.040406 (+ 6153us) server_negotiation.cc:407] Connection header received
0926 20:25:16.040579 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:16.040582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:16.040631 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:16.040681 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:16.043697 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:16.044456 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:16.046778 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:16.047134 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:16.047207 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:16.050954 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:16.050958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:16.050962 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:16.050990 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:16.052696 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:16.053200 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:16.053203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:16.053205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:16.053289 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:16.053568 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:16.053571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:16.053573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:16.053779 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:16.053925 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:16.054481 (+ 556us) server_negotiation.cc:299] Negotiation successful
0926 20:25:16.054603 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:25:16.065330 22851 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1cdadadc4cf14d8abaed6f09dcebc343 [term 1 FOLLOWER]: Refusing update from remote peer 5400d8c430994f5397171a981e6392ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:25:16.065389 22926 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b27cc4c6099436fb7e6b08c16d79ac9 [term 1 FOLLOWER]: Refusing update from remote peer 5400d8c430994f5397171a981e6392ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:25:16.066205 23002 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b27cc4c6099436fb7e6b08c16d79ac9" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:25:16.066674 23002 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cdadadc4cf14d8abaed6f09dcebc343" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:25:16.070402 22975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cdadadc4cf14d8abaed6f09dcebc343 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5400d8c430994f5397171a981e6392ca. Latest consensus state: current_term: 1 leader_uuid: "5400d8c430994f5397171a981e6392ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5400d8c430994f5397171a981e6392ca" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36523 } } peers { permanent_uuid: "1cdadadc4cf14d8abaed6f09dcebc343" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } } peers { permanent_uuid: "3b27cc4c6099436fb7e6b08c16d79ac9" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } } }
I20260926 20:25:16.070540 22975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cdadadc4cf14d8abaed6f09dcebc343 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:25:16.070693 22974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b27cc4c6099436fb7e6b08c16d79ac9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5400d8c430994f5397171a981e6392ca. Latest consensus state: current_term: 1 leader_uuid: "5400d8c430994f5397171a981e6392ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5400d8c430994f5397171a981e6392ca" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36523 } } peers { permanent_uuid: "1cdadadc4cf14d8abaed6f09dcebc343" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } } peers { permanent_uuid: "3b27cc4c6099436fb7e6b08c16d79ac9" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } } }
I20260926 20:25:16.070806 22974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b27cc4c6099436fb7e6b08c16d79ac9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:25:16.078255 22959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5400d8c430994f5397171a981e6392ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5400d8c430994f5397171a981e6392ca" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36523 } } peers { permanent_uuid: "1cdadadc4cf14d8abaed6f09dcebc343" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } } peers { permanent_uuid: "3b27cc4c6099436fb7e6b08c16d79ac9" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } } }
I20260926 20:25:16.078392 22959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:16.079418 22974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b27cc4c6099436fb7e6b08c16d79ac9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5400d8c430994f5397171a981e6392ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5400d8c430994f5397171a981e6392ca" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36523 } } peers { permanent_uuid: "1cdadadc4cf14d8abaed6f09dcebc343" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } } peers { permanent_uuid: "3b27cc4c6099436fb7e6b08c16d79ac9" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } } }
I20260926 20:25:16.079566 22974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b27cc4c6099436fb7e6b08c16d79ac9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:25:16.079967 23010 mvcc.cc:205] Tried to move back new op lower bound from 7333700878596448256 to 7333700878452727808. Current Snapshot: MvccSnapshot[applied={T|T < 7333700878596448256}]
I20260926 20:25:16.080144 23012 mvcc.cc:205] Tried to move back new op lower bound from 7333700878596448256 to 7333700878452727808. Current Snapshot: MvccSnapshot[applied={T|T < 7333700878596448256}]
I20260926 20:25:16.082894 23002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5400d8c430994f5397171a981e6392ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5400d8c430994f5397171a981e6392ca" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36523 } } peers { permanent_uuid: "1cdadadc4cf14d8abaed6f09dcebc343" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } } peers { permanent_uuid: "3b27cc4c6099436fb7e6b08c16d79ac9" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } } }
I20260926 20:25:16.083032 23002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:16.083920 22975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cdadadc4cf14d8abaed6f09dcebc343 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5400d8c430994f5397171a981e6392ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5400d8c430994f5397171a981e6392ca" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 36523 } } peers { permanent_uuid: "1cdadadc4cf14d8abaed6f09dcebc343" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 40121 } } peers { permanent_uuid: "3b27cc4c6099436fb7e6b08c16d79ac9" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 36867 } } }
I20260926 20:25:16.084041 22975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cdadadc4cf14d8abaed6f09dcebc343 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:25:16.084159 23013 mvcc.cc:205] Tried to move back new op lower bound from 7333700878596448256 to 7333700878452727808. Current Snapshot: MvccSnapshot[applied={T|T < 7333700878596448256}]
I20260926 20:25:16.089712 23005 catalog_manager.cc:1347] Generated new cluster ID: 4c636e09e88d420e85a4f5a720371e42
I20260926 20:25:16.089793 23005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:25:16.106041 23005 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:25:16.107375 23005 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:25:16.127791 23005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5400d8c430994f5397171a981e6392ca: Generated new TSK 0
I20260926 20:25:16.128834 23005 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:25:16.164623 22879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:16.161769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55504 (local address 127.0.116.253:40121)
0926 20:25:16.161898 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:16.161903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:16.163270 (+ 1367us) server_negotiation.cc:407] Connection header received
0926 20:25:16.163475 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:16.163478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:16.163565 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:16.163621 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:16.164485 (+ 864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55504: BlockingRecv error: recv got EOF from 127.0.0.1:55504 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 20:25:16.165982 22805 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:16.162632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42026 (local address 127.0.116.254:36523)
0926 20:25:16.162746 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:16.162751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:16.165134 (+ 2383us) server_negotiation.cc:407] Connection header received
0926 20:25:16.165300 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:16.165303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:16.165351 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:16.165399 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:16.165865 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42026: BlockingRecv error: recv got EOF from 127.0.0.1:42026 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 20:25:16.167861 22957 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:16.166448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47674 (local address 127.0.116.252:36867)
0926 20:25:16.166585 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:16.166589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:16.166908 (+ 319us) server_negotiation.cc:407] Connection header received
0926 20:25:16.167073 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:16.167076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:16.167121 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:16.167168 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:16.167746 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47674: BlockingRecv error: recv got EOF from 127.0.0.1:47674 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 20:25:16.172179 22879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:16.170744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55514 (local address 127.0.116.253:40121)
0926 20:25:16.170867 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:16.170873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:16.171169 (+ 296us) server_negotiation.cc:407] Connection header received
0926 20:25:16.171383 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:16.171386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:16.171461 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:16.171551 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:16.172072 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55514: BlockingRecv error: recv got EOF from 127.0.0.1:55514 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 20:25:16.177264 22879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:16.175553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55522 (local address 127.0.116.253:40121)
0926 20:25:16.175695 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:16.175698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:16.176402 (+ 704us) server_negotiation.cc:407] Connection header received
0926 20:25:16.176581 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:16.176584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:16.176633 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:16.176682 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:16.177163 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55522: BlockingRecv error: recv got EOF from 127.0.0.1:55522 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260926 20:25:16.182781 22879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:16.181227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55532 (local address 127.0.116.253:40121)
0926 20:25:16.181330 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:16.181335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:16.181873 (+ 538us) server_negotiation.cc:407] Connection header received
0926 20:25:16.182083 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:16.182087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:16.182136 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:16.182187 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:16.182665 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55532: BlockingRecv error: recv got EOF from 127.0.0.1:55532 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 20:25:16.190591 22879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:16.188922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55544 (local address 127.0.116.253:40121)
0926 20:25:16.189071 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:16.189075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:16.189626 (+ 551us) server_negotiation.cc:407] Connection header received
0926 20:25:16.189821 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:16.189824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:16.189879 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:16.189932 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:16.190476 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55544: BlockingRecv error: recv got EOF from 127.0.0.1:55544 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 20:25:16.203938 22879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:16.201962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55546 (local address 127.0.116.253:40121)
0926 20:25:16.202106 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:16.202110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:16.202778 (+ 668us) server_negotiation.cc:407] Connection header received
0926 20:25:16.203025 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:16.203029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:16.203091 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:16.203153 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:16.203810 (+ 657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55546: BlockingRecv error: recv got EOF from 127.0.0.1:55546 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 20:25:16.224907 22879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:16.222701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55560 (local address 127.0.116.253:40121)
0926 20:25:16.222832 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:16.222837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:16.223651 (+ 814us) server_negotiation.cc:407] Connection header received
0926 20:25:16.223940 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:16.223944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:16.224029 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:16.224098 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:16.224777 (+ 679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55560: BlockingRecv error: recv got EOF from 127.0.0.1:55560 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 20:25:16.260808 22879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:16.259527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55562 (local address 127.0.116.253:40121)
0926 20:25:16.259667 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:16.259672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:16.259897 (+ 225us) server_negotiation.cc:407] Connection header received
0926 20:25:16.260081 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:16.260084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:16.260142 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:16.260233 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:16.260683 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55562: BlockingRecv error: recv got EOF from 127.0.0.1:55562 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 20:25:16.328243 22879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:16.326988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55572 (local address 127.0.116.253:40121)
0926 20:25:16.327127 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:16.327131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:16.327293 (+ 162us) server_negotiation.cc:407] Connection header received
0926 20:25:16.327427 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:16.327431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:16.327590 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:16.327676 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:16.328137 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55572: BlockingRecv error: recv got EOF from 127.0.0.1:55572 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 20:25:16.459846 22879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:16.458499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55578 (local address 127.0.116.253:40121)
0926 20:25:16.458636 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:16.458640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:16.458891 (+ 251us) server_negotiation.cc:407] Connection header received
0926 20:25:16.459052 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:16.459055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:16.459112 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:16.459193 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:16.459709 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55578: BlockingRecv error: recv got EOF from 127.0.0.1:55578 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 20:25:16.719641 22879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:16.718400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55592 (local address 127.0.116.253:40121)
0926 20:25:16.718571 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:16.718576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:16.718594 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:25:16.718744 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:16.718747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:16.718796 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:16.718916 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:16.719479 (+ 563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55592: BlockingRecv error: recv got EOF from 127.0.0.1:55592 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 20:25:17.031405 23001 catalog_manager.cc:1259] Loaded cluster ID: 4c636e09e88d420e85a4f5a720371e42
I20260926 20:25:17.031525 23001 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3b27cc4c6099436fb7e6b08c16d79ac9: loading cluster ID for follower catalog manager: success
I20260926 20:25:17.033200 23001 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3b27cc4c6099436fb7e6b08c16d79ac9: acquiring CA information for follower catalog manager: success
I20260926 20:25:17.034368 23001 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3b27cc4c6099436fb7e6b08c16d79ac9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:25:17.046823 23000 catalog_manager.cc:1259] Loaded cluster ID: 4c636e09e88d420e85a4f5a720371e42
I20260926 20:25:17.046911 23000 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1cdadadc4cf14d8abaed6f09dcebc343: loading cluster ID for follower catalog manager: success
I20260926 20:25:17.048568 23000 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1cdadadc4cf14d8abaed6f09dcebc343: acquiring CA information for follower catalog manager: success
I20260926 20:25:17.049981 23000 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1cdadadc4cf14d8abaed6f09dcebc343: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:25:17.237493 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:17.233714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55608 (local address 127.0.116.253:40121)
0926 20:25:17.233955 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:17.233959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:17.233974 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:25:17.234104 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:17.234106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:17.234228 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:17.234322 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:25:17.234561 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:17.235095 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:17.236031 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:17.236252 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:17.236331 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:17.236623 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:25:17.237094 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:25:17.237187 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:17.237239 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 20:25:17.237288 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":76,"threads_started":1}
I20260926 20:25:17.244427 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:17.240678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47686 (local address 127.0.116.252:36867)
0926 20:25:17.240911 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:17.240915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:17.240927 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:25:17.241059 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:17.241061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:17.241174 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:17.241280 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:25:17.241478 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:17.242006 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:17.242792 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:17.243054 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:17.243135 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:17.243596 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:25:17.244088 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:25:17.244196 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:17.244253 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 20:25:17.244304 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":95,"threads_started":1}
I20260926 20:25:17.251768 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:17.247847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47696 (local address 127.0.116.252:36867)
0926 20:25:17.247964 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:17.247970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:17.248058 (+ 88us) server_negotiation.cc:407] Connection header received
0926 20:25:17.248219 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:17.248223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:17.248274 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:17.248325 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:25:17.248703 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:17.249450 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:17.250556 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:17.250894 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:17.250983 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:17.251223 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:25:17.251299 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:25:17.251379 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:17.251537 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 20:25:17.251599 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:25:17.251784 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:17.247529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55622 (local address 127.0.116.253:40121)
0926 20:25:17.247911 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:17.247916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:17.247943 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:25:17.247998 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:17.248001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:17.248057 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:17.248134 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:25:17.248388 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:17.249194 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:17.250510 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:17.250813 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:17.250935 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:17.251261 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:25:17.251330 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:25:17.251376 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:17.251588 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 20:25:17.251648 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260926 20:25:17.251792 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:17.247681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42036 (local address 127.0.116.254:36523)
0926 20:25:17.248110 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:17.248116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:17.248129 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:25:17.248225 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:17.248228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:17.248398 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:17.248537 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:25:17.248745 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:17.249311 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:17.250254 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:17.250567 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:17.250664 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:17.250997 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:25:17.251368 (+ 371us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:25:17.251480 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:17.251587 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 20:25:17.251648 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":70,"threads_started":1}
I20260926 20:25:17.255144 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 22734
I20260926 20:25:17.261539 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 22809
I20260926 20:25:17.267715 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 22884
2026-09-26T20:25:17Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3135 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:25:17 dist-test-slave-qsm9 krb5kdc[23099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:25:17 dist-test-slave-qsm9 krb5kdc[23099](info): set up 2 sockets
Sep 26 20:25:17 dist-test-slave-qsm9 krb5kdc[23099](info): commencing operation
krb5kdc: starting...
W20260926 20:25:18.360494 467 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/test-user.keytab.
Sep 26 20:25:18 dist-test-slave-qsm9 krb5kdc[23099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:25:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:25:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
Entry for principal kudu/127.0.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab.
I20260926 20:25:18.492148 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:34085
--webserver_interface=127.0.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:34085,127.0.116.253:35117,127.0.116.252:37071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:25:18.603667 23112 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:25:18.603976 23112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:25:18.603999 23112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:25:18.604039 23112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:25:18.608228 23112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:25:18.608295 23112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:25:18.608315 23112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:25:18.608332 23112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:25:18.608347 23112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:25:18.613804 23112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:34085,127.0.116.253:35117,127.0.116.252:37071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/kudu_127.0.116.254.keytab
--principal=kudu/127.0.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.254:34085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:25:18.615136 23112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:25:18.616247 23112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:25:18.624818 23120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:18.624989 23118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:18.625110 23117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:18.625385 23112 server_base.cc:1034] running on GCE node
I20260926 20:25:18.625944 23112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:25:18.627149 23112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:25:18.628336 23112 hybrid_clock.cc:648] HybridClock initialized: now 1790454318628321 us; error 27 us; skew 500 ppm
Sep 26 20:25:18 dist-test-slave-qsm9 krb5kdc[23099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:25:18.631143 23112 init.cc:377] Logged in from keytab as kudu/127.0.116.254@KRBTEST.COM (short username kudu)
I20260926 20:25:18.632066 23112 webserver.cc:460] Webserver started at http://127.0.116.254:33573/ using document root <none> and password file <none>
I20260926 20:25:18.632584 23112 fs_manager.cc:359] Metadata directory not provided
I20260926 20:25:18.632655 23112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:25:18.632817 23112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:25:18.634863 23112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-0/data/instance:
uuid: "816f68eed11341508a286efe1aaa7277"
format_stamp: "Formatted at 2026-09-26 20:25:18 on dist-test-slave-qsm9"
server_key: "c8d2503ff29453acf17e70d0c4242ba6"
server_key_iv: "df9378f8e1c2cb9729b55b3a3ed66f74"
server_key_version: "encryptionkey@0"
I20260926 20:25:18.635381 23112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-0/wal/instance:
uuid: "816f68eed11341508a286efe1aaa7277"
format_stamp: "Formatted at 2026-09-26 20:25:18 on dist-test-slave-qsm9"
server_key: "c8d2503ff29453acf17e70d0c4242ba6"
server_key_iv: "df9378f8e1c2cb9729b55b3a3ed66f74"
server_key_version: "encryptionkey@0"
I20260926 20:25:18.639180 23112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260926 20:25:18.641754 23127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:25:18.643055 23112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:25:18.643204 23112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-0/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-0/wal
uuid: "816f68eed11341508a286efe1aaa7277"
format_stamp: "Formatted at 2026-09-26 20:25:18 on dist-test-slave-qsm9"
server_key: "c8d2503ff29453acf17e70d0c4242ba6"
server_key_iv: "df9378f8e1c2cb9729b55b3a3ed66f74"
server_key_version: "encryptionkey@0"
I20260926 20:25:18.643321 23112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:25:18.654215 23112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:25:18.657418 23112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:25:18.657641 23112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:25:18.664891 23112 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.254:34085
I20260926 20:25:18.665179 23179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.254:34085 every 8 connection(s)
I20260926 20:25:18.665954 23112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-0/data/info.pb
I20260926 20:25:18.668879 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 23112
I20260926 20:25:18.668964 23180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:25:18.669133 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-0/wal/instance
I20260926 20:25:18.669488 467 external_mini_cluster.cc:1428] Setting key e2f87a15d8be7986db545afaee0e018c
I20260926 20:25:18.673554 23180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34085 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.116.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.253@KRBTEST.COM" created.
Sep 26 20:25:18 dist-test-slave-qsm9 krb5kdc[23099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:25:18.689539 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:18.675976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57774 (local address 127.0.116.254:34085)
0926 20:25:18.676546 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:18.676559 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:18.676608 (+ 49us) server_negotiation.cc:407] Connection header received
0926 20:25:18.677670 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:18.677674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:18.677862 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:18.678057 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:18.678941 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:18.679964 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:18.681030 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:18.681398 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:18.681570 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:18.683947 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:18.683953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:18.683962 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:18.683982 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:18.685930 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:18.686508 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:18.686515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:18.686517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:18.686636 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:18.687034 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:18.687041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:18.687042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:18.687464 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:18.687694 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:18.688075 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 20:25:18.688353 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"spinlock_wait_cycles":28928,"thread_start_us":148,"threads_started":1}
I20260926 20:25:18.689535 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:18.675975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:34085 (local address 127.0.0.1:57774)
0926 20:25:18.676470 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:18.676521 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:25:18.677336 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:25:18.678053 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:25:18.678058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:25:18.678496 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:25:18.678789 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:18.678803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:18.680081 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:18.680085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:18.680912 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:18.680921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:18.681548 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:18.681552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:18.681682 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:18.682282 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:25:18.682299 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:25:18.683795 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:25:18.686081 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:18.686092 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:18.686103 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:18.686381 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:18.686740 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:18.686746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:18.686748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:18.686922 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:18.687658 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:25:18.687665 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:25:18.687820 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 20:25:18.688013 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 20:25:18.688408 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":120,"thread_start_us":143,"threads_started":1}
I20260926 20:25:18.692250 23180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:25:18.693030 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:18.692719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:35117 (local address 127.0.0.1:48592)
0926 20:25:18.692871 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:18.692960 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.253:35117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 20:25:18.693222 23128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.116.253:35117: connect: Connection refused (error 111)
I20260926 20:25:18.694168 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:18.693947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:35117 (local address 127.0.0.1:48596)
0926 20:25:18.694062 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:18.694114 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.253:35117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 20:25:18.694336 23180 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.253:35117: Network error: Client connection negotiation failed: client connection to 127.0.116.253:35117: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/kudu_127.0.116.253.keytab.
Entry for principal kudu/127.0.116.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/kudu_127.0.116.253.keytab.
I20260926 20:25:18.698149 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.253:35117
--webserver_interface=127.0.116.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/kudu_127.0.116.253.keytab
--principal=kudu/127.0.116.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:34085,127.0.116.253:35117,127.0.116.252:37071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:25:18.743654 23180 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } attempt: 1
I20260926 20:25:18.744656 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:18.744272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:35117 (local address 127.0.0.1:48608)
0926 20:25:18.744488 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:18.744571 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.253:35117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260926 20:25:18.745455 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:18.745206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:35117 (local address 127.0.0.1:48622)
0926 20:25:18.745342 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:18.745391 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.253:35117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 20:25:18.745787 23180 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.253:35117: Network error: Client connection negotiation failed: client connection to 127.0.116.253:35117: connect: Connection refused (error 111)
W20260926 20:25:18.812429 23187 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:25:18.812731 23187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:25:18.812755 23187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:25:18.812848 23187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 20:25:18.813987 23180 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } attempt: 2
I20260926 20:25:18.814901 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:18.814554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:35117 (local address 127.0.0.1:48636)
0926 20:25:18.814740 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:18.814812 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.253:35117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 20:25:18.815840 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:18.815598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:35117 (local address 127.0.0.1:48640)
0926 20:25:18.815716 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:18.815776 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.253:35117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 20:25:18.815984 23180 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.253:35117: Network error: Client connection negotiation failed: client connection to 127.0.116.253:35117: connect: Connection refused (error 111)
W20260926 20:25:18.817498 23187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:25:18.817554 23187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:25:18.817574 23187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:25:18.817592 23187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:25:18.817646 23187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:25:18.823609 23187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:34085,127.0.116.253:35117,127.0.116.252:37071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/kudu_127.0.116.253.keytab
--principal=kudu/127.0.116.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.253:35117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.116.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:25:18.824769 23187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:25:18.825865 23187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:25:18.834736 23195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:18.834734 23192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:18.834730 23193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:18.834933 23187 server_base.cc:1034] running on GCE node
I20260926 20:25:18.835671 23187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:25:18.836839 23187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:25:18.838004 23187 hybrid_clock.cc:648] HybridClock initialized: now 1790454318837990 us; error 33 us; skew 500 ppm
Sep 26 20:25:18 dist-test-slave-qsm9 krb5kdc[23099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:25:18.840989 23187 init.cc:377] Logged in from keytab as kudu/127.0.116.253@KRBTEST.COM (short username kudu)
I20260926 20:25:18.841976 23187 webserver.cc:460] Webserver started at http://127.0.116.253:39153/ using document root <none> and password file <none>
I20260926 20:25:18.842551 23187 fs_manager.cc:359] Metadata directory not provided
I20260926 20:25:18.842643 23187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:25:18.842881 23187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:25:18.845117 23187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-1/data/instance:
uuid: "65b76032762f484aa93ec99066eebe49"
format_stamp: "Formatted at 2026-09-26 20:25:18 on dist-test-slave-qsm9"
server_key: "0d6f8941025256bb1227381381cea73e"
server_key_iv: "14116886ab36af915bd931c6a97a3c2b"
server_key_version: "encryptionkey@0"
I20260926 20:25:18.845700 23187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-1/wal/instance:
uuid: "65b76032762f484aa93ec99066eebe49"
format_stamp: "Formatted at 2026-09-26 20:25:18 on dist-test-slave-qsm9"
server_key: "0d6f8941025256bb1227381381cea73e"
server_key_iv: "14116886ab36af915bd931c6a97a3c2b"
server_key_version: "encryptionkey@0"
I20260926 20:25:18.850057 23187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:25:18.852772 23202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:25:18.854184 23187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:25:18.854353 23187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-1/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-1/wal
uuid: "65b76032762f484aa93ec99066eebe49"
format_stamp: "Formatted at 2026-09-26 20:25:18 on dist-test-slave-qsm9"
server_key: "0d6f8941025256bb1227381381cea73e"
server_key_iv: "14116886ab36af915bd931c6a97a3c2b"
server_key_version: "encryptionkey@0"
I20260926 20:25:18.854511 23187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:25:18.875177 23187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:25:18.878731 23187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:25:18.878957 23187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:25:18.886375 23187 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.253:35117
I20260926 20:25:18.886705 23254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.253:35117 every 8 connection(s)
I20260926 20:25:18.887409 23187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-1/data/info.pb
I20260926 20:25:18.890448 23255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:25:18.894606 23255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34085 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:25:18.895301 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 23187
I20260926 20:25:18.895455 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-1/wal/instance
I20260926 20:25:18.895768 467 external_mini_cluster.cc:1428] Setting key 2745a36b28787c91380d1239abe48d14
WARNING: no policy specified for kudu/127.0.116.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.116.252@KRBTEST.COM" created.
I20260926 20:25:18.907208 23180 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } attempt: 3
Sep 26 20:25:18 dist-test-slave-qsm9 krb5kdc[23099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.253@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:25:18.912994 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:18.896840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57782 (local address 127.0.116.254:34085)
0926 20:25:18.896996 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:18.897000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:18.898020 (+ 1020us) server_negotiation.cc:407] Connection header received
0926 20:25:18.898976 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:18.898983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:18.899058 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:18.899182 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:18.900457 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:18.901396 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:18.902563 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:18.902992 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:18.903129 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:18.908702 (+ 5573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:18.908712 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:18.908715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:18.908739 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:18.910349 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:18.911050 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:18.911053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:18.911055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:18.911151 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:18.911630 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:18.911637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:18.911639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:18.911890 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:18.912008 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:18.912691 (+ 683us) server_negotiation.cc:299] Negotiation successful
0926 20:25:18.912830 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57,"spinlock_wait_cycles":3968}
Sep 26 20:25:18 dist-test-slave-qsm9 krb5kdc[23099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for kudu/127.0.116.253@KRBTEST.COM
I20260926 20:25:18.914520 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:18.897295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:34085 (local address 127.0.0.1:57782)
0926 20:25:18.897840 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:18.897893 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:25:18.898759 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:25:18.899338 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:25:18.899344 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:25:18.899778 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:25:18.900286 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:18.900301 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:18.901547 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:18.901550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:18.902428 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:18.902438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:18.903187 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:18.903191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:18.903364 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:18.904047 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:25:18.904063 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:25:18.908519 (+ 4456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:25:18.910489 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:18.910496 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:18.910521 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:18.910945 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:18.911257 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:18.911263 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:18.911268 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:18.911470 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:18.912019 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:25:18.912028 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:25:18.912374 (+ 346us) client_negotiation.cc:769] Sending connection context
0926 20:25:18.913173 (+ 799us) client_negotiation.cc:240] Negotiation successful
0926 20:25:18.913564 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":164,"threads_started":1}
I20260926 20:25:18.917268 23255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:25:18.919142 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:18.908061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48646 (local address 127.0.116.253:35117)
0926 20:25:18.908473 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:18.908488 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:18.908517 (+ 29us) server_negotiation.cc:407] Connection header received
0926 20:25:18.908570 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:18.908579 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:18.908819 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:18.908968 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:18.909453 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:18.910692 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:18.911689 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:18.912374 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:18.912504 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:18.915076 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:18.915086 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:18.915100 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:18.915126 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:18.916986 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:18.917503 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:18.917510 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:18.917511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:18.917600 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:18.917910 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:18.917913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:18.917915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:18.918227 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:18.918367 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:18.918817 (+ 450us) server_negotiation.cc:299] Negotiation successful
0926 20:25:18.918942 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":130,"threads_started":1}
I20260926 20:25:18.919147 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:18.907786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:35117 (local address 127.0.0.1:48646)
0926 20:25:18.907972 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:18.907990 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:25:18.908087 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:25:18.908980 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:25:18.908984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:25:18.909171 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:25:18.909328 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:18.909337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:18.910816 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:18.910821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:18.911568 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:18.911577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:18.912498 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:18.912502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:18.912629 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:18.913355 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:25:18.913369 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:25:18.914914 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:25:18.917115 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:18.917118 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:18.917121 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:18.917374 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:18.917686 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:18.917690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:18.917692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:18.917803 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:18.918377 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:25:18.918379 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:25:18.918619 (+ 240us) client_negotiation.cc:769] Sending connection context
0926 20:25:18.918746 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 20:25:18.918906 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 20:25:18.921226 23180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:25:18.922118 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:18.921853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:37071 (local address 127.0.0.1:50810)
0926 20:25:18.921968 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:18.922017 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:37071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 20:25:18.922942 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:18.922639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:37071 (local address 127.0.0.1:50824)
0926 20:25:18.922821 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:18.922875 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:37071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
Sep 26 20:25:18 dist-test-slave-qsm9 krb5kdc[23099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.253@KRBTEST.COM for kudu/127.0.116.253@KRBTEST.COM
W20260926 20:25:18.923127 23180 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.252:37071: Network error: Client connection negotiation failed: client connection to 127.0.116.252:37071: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.116.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/kudu_127.0.116.252.keytab.
Entry for principal kudu/127.0.116.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/kudu_127.0.116.252.keytab.
I20260926 20:25:18.927416 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:18.917863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:35117 (local address 127.0.0.1:48654)
0926 20:25:18.918061 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:18.918077 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:25:18.918157 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:25:18.918617 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:25:18.918621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:25:18.918806 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:25:18.918942 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:18.918949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:18.920076 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:18.920082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:18.920698 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:18.920707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:18.921206 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:18.921209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:18.921302 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:18.921824 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:25:18.921839 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:25:18.923374 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:25:18.925807 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:18.925810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:18.925812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:18.926027 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:18.926300 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:18.926303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:18.926305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:18.926430 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:18.926885 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:25:18.926889 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:25:18.927010 (+ 121us) client_negotiation.cc:769] Sending connection context
0926 20:25:18.927105 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 20:25:18.927286 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260926 20:25:18.927721 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:18.917900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.0.116.253:35117)
0926 20:25:18.918241 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:18.918249 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:18.918263 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:25:18.918300 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:18.918306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:18.918350 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:18.918431 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:18.919061 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:18.919960 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:18.920833 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:18.921087 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:18.921198 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:18.923580 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:18.923585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:18.923589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:18.923618 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:18.925682 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:18.926125 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:18.926128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:18.926129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:18.926205 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:18.926526 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:18.926529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:18.926530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:18.926717 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:18.926796 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:18.927222 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 20:25:18.927391 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":115,"threads_started":1}
I20260926 20:25:18.928335 467 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3m9pMa/build/debug/bin/kudu
/tmp/dist-test-task3m9pMa/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.252:37071
--webserver_interface=127.0.116.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/kudu_127.0.116.252.keytab
--principal=kudu/127.0.116.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.116.212:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.116.254:34085,127.0.116.253:35117,127.0.116.252:37071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:25:18.928468 23255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:25:18.929325 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:18.928985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:37071 (local address 127.0.0.1:50840)
0926 20:25:18.929133 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:18.929241 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:37071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 20:25:18.929587 23204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.116.252:37071: connect: Connection refused (error 111)
I20260926 20:25:18.930565 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:18.930298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:37071 (local address 127.0.0.1:50850)
0926 20:25:18.930431 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:18.930493 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:37071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 20:25:18.930802 23255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.252:37071: Network error: Client connection negotiation failed: client connection to 127.0.116.252:37071: connect: Connection refused (error 111)
I20260926 20:25:18.954308 23180 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } attempt: 1
I20260926 20:25:18.955365 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:18.954971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:37071 (local address 127.0.0.1:50854)
0926 20:25:18.955156 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:18.955244 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:37071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 20:25:18.956240 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:18.955983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:37071 (local address 127.0.0.1:50860)
0926 20:25:18.956101 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:18.956163 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:37071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 20:25:18.956453 23180 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.252:37071: Network error: Client connection negotiation failed: client connection to 127.0.116.252:37071: connect: Connection refused (error 111)
I20260926 20:25:18.980121 23255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } attempt: 1
I20260926 20:25:18.981091 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:18.980757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:37071 (local address 127.0.0.1:50874)
0926 20:25:18.980936 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:18.981006 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:37071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 20:25:18.982002 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:18.981775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:37071 (local address 127.0.0.1:50884)
0926 20:25:18.981891 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:18.981941 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:37071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 20:25:18.982177 23255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.252:37071: Network error: Client connection negotiation failed: client connection to 127.0.116.252:37071: connect: Connection refused (error 111)
I20260926 20:25:19.031634 23180 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } attempt: 2
I20260926 20:25:19.032553 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:19.032215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:37071 (local address 127.0.0.1:50886)
0926 20:25:19.032379 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:19.032443 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:37071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 20:25:19.033316 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:19.033105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:37071 (local address 127.0.0.1:50900)
0926 20:25:19.033210 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:19.033264 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:37071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 20:25:19.033452 23180 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.252:37071: Network error: Client connection negotiation failed: client connection to 127.0.116.252:37071: connect: Connection refused (error 111)
W20260926 20:25:19.048163 23262 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:25:19.048532 23262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:25:19.048601 23262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:25:19.048693 23262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 20:25:19.050390 23255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } attempt: 2
I20260926 20:25:19.051283 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:19.050961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:37071 (local address 127.0.0.1:50914)
0926 20:25:19.051137 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:19.051203 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:37071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 20:25:19.052060 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:19.051833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:37071 (local address 127.0.0.1:50928)
0926 20:25:19.051953 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:19.052006 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.116.252:37071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 20:25:19.052176 23255 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.116.252:37071: Network error: Client connection negotiation failed: client connection to 127.0.116.252:37071: connect: Connection refused (error 111)
W20260926 20:25:19.053334 23262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:25:19.053416 23262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:25:19.053464 23262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:25:19.053507 23262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:25:19.053550 23262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:25:19.059324 23262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.116.212:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.116.254:34085,127.0.116.253:35117,127.0.116.252:37071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/krb5kdc/kudu_127.0.116.252.keytab
--principal=kudu/127.0.116.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.116.252:37071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.116.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:25:19.060614 23262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:25:19.061992 23262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:25:19.071199 23269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:19.071161 23271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:25:19.071223 23268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:25:19.071404 23262 server_base.cc:1034] running on GCE node
I20260926 20:25:19.072176 23262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:25:19.073383 23262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:25:19.074568 23262 hybrid_clock.cc:648] HybridClock initialized: now 1790454319074535 us; error 43 us; skew 500 ppm
Sep 26 20:25:19 dist-test-slave-qsm9 krb5kdc[23099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:25:19.077672 23262 init.cc:377] Logged in from keytab as kudu/127.0.116.252@KRBTEST.COM (short username kudu)
I20260926 20:25:19.078648 23262 webserver.cc:460] Webserver started at http://127.0.116.252:36129/ using document root <none> and password file <none>
I20260926 20:25:19.079221 23262 fs_manager.cc:359] Metadata directory not provided
I20260926 20:25:19.079312 23262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:25:19.079609 23262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:25:19.081670 23262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-2/data/instance:
uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1"
format_stamp: "Formatted at 2026-09-26 20:25:19 on dist-test-slave-qsm9"
server_key: "b273570fa055b75b6deff672b0ceb142"
server_key_iv: "294e7b2a51ac82b8bbf738ca2c4bebb7"
server_key_version: "encryptionkey@0"
I20260926 20:25:19.082273 23262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-2/wal/instance:
uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1"
format_stamp: "Formatted at 2026-09-26 20:25:19 on dist-test-slave-qsm9"
server_key: "b273570fa055b75b6deff672b0ceb142"
server_key_iv: "294e7b2a51ac82b8bbf738ca2c4bebb7"
server_key_version: "encryptionkey@0"
I20260926 20:25:19.086287 23262 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:25:19.088944 23278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:25:19.090209 23262 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:25:19.090353 23262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-2/data,/tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-2/wal
uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1"
format_stamp: "Formatted at 2026-09-26 20:25:19 on dist-test-slave-qsm9"
server_key: "b273570fa055b75b6deff672b0ceb142"
server_key_iv: "294e7b2a51ac82b8bbf738ca2c4bebb7"
server_key_version: "encryptionkey@0"
I20260926 20:25:19.090476 23262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:25:19.104259 23262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:25:19.107781 23262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:25:19.107992 23262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:25:19.115044 23262 rpc_server.cc:307] RPC server started. Bound to: 127.0.116.252:37071
I20260926 20:25:19.115451 23330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.116.252:37071 every 8 connection(s)
I20260926 20:25:19.116098 23262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-2/data/info.pb
I20260926 20:25:19.119158 23331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:25:19.123027 23331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34085 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:25:19.125757 467 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3m9pMa/build/debug/bin/kudu as pid 23262
I20260926 20:25:19.125988 467 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3m9pMa/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790454033974695-467-0/minicluster-data/master-2/wal/instance
I20260926 20:25:19.126281 467 external_mini_cluster.cc:1428] Setting key 98597d258a7f9d7147c5dc589ae49b68
I20260926 20:25:19.132622 23180 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } attempt: 3
Sep 26 20:25:19 dist-test-slave-qsm9 krb5kdc[23099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.252@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
Sep 26 20:25:19 dist-test-slave-qsm9 krb5kdc[23099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.252@KRBTEST.COM
I20260926 20:25:19.137817 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:19.125095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.0.116.254:34085)
0926 20:25:19.125287 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.125292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.126096 (+ 804us) server_negotiation.cc:407] Connection header received
0926 20:25:19.127389 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.127392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.127455 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.127615 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.128744 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.129246 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.130257 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.130551 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.130663 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:19.133629 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:19.133634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:19.133637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:19.133662 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:19.135340 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:19.136024 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:19.136028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:19.136030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:19.136124 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:19.136477 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:19.136481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:19.136483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:19.136707 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:19.136795 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:19.137515 (+ 720us) server_negotiation.cc:299] Negotiation successful
0926 20:25:19.137652 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
Sep 26 20:25:19 dist-test-slave-qsm9 krb5kdc[23099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.254@KRBTEST.COM for kudu/127.0.116.252@KRBTEST.COM
I20260926 20:25:19.139192 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:19.125456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.254:34085 (local address 127.0.0.1:57790)
0926 20:25:19.125943 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:19.125996 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:25:19.127146 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:25:19.127701 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:25:19.127712 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:25:19.128110 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:25:19.128590 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:19.128610 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.129366 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.129370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:19.130122 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:19.130131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.130705 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.130709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:19.130887 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:19.131468 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:25:19.131485 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:25:19.133424 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:25:19.135452 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:19.135463 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:19.135475 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:19.135912 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:19.136235 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:19.136237 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:19.136238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:19.136370 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:19.136845 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:25:19.136853 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:25:19.137200 (+ 347us) client_negotiation.cc:769] Sending connection context
0926 20:25:19.138027 (+ 827us) client_negotiation.cc:240] Negotiation successful
0926 20:25:19.138186 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":53,"thread_start_us":144,"threads_started":1}
I20260926 20:25:19.139192 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:19.126988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50936 (local address 127.0.116.252:37071)
0926 20:25:19.127326 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.127339 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.127390 (+ 51us) server_negotiation.cc:407] Connection header received
0926 20:25:19.127644 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.127649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.127822 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.128004 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.128814 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.129661 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.130491 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.130849 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.130965 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:19.133976 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:19.133983 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:19.134004 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:19.134025 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:19.136062 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:19.136568 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:19.136577 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:19.136580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:19.136670 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:19.136959 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:19.136964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:19.136966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:19.137281 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:19.137500 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:19.137740 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 20:25:19.138071 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":233,"spinlock_wait_cycles":768,"thread_start_us":158,"threads_started":1}
I20260926 20:25:19.140828 23331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:25:19.143420 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:19.133146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:37071 (local address 127.0.0.1:50948)
0926 20:25:19.133368 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:19.133385 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:25:19.133462 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:25:19.133869 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:25:19.133872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:25:19.134384 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:25:19.134514 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:19.134525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.135647 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.135651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:19.136469 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:19.136484 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.137057 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.137060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:19.137142 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:19.137615 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:25:19.137634 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:25:19.139040 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:25:19.141743 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:19.141748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:19.141750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:19.141970 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:19.142295 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:19.142299 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:19.142304 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:19.142413 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:19.142860 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:25:19.142863 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:25:19.142978 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 20:25:19.143072 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 20:25:19.143223 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260926 20:25:19.143540 23255 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } attempt: 3
I20260926 20:25:19.144071 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:19.133207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50948 (local address 127.0.116.252:37071)
0926 20:25:19.133669 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.133676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.133686 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:25:19.133712 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.133714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.133746 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.133846 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.134664 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.135526 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.136578 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.136923 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.137049 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:19.139174 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:19.139178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:19.139180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:19.139202 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:19.141649 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:19.142073 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:19.142078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:19.142079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:19.142184 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:19.142533 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:19.142536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:19.142537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:19.142747 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:19.143723 (+ 976us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:19.143795 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 20:25:19.143905 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":108,"threads_started":1}
Sep 26 20:25:19 dist-test-slave-qsm9 krb5kdc[23099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.252@KRBTEST.COM for kudu/127.0.116.253@KRBTEST.COM
Sep 26 20:25:19 dist-test-slave-qsm9 krb5kdc[23099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.254@KRBTEST.COM
I20260926 20:25:19.148949 23180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277: Bootstrap starting.
Sep 26 20:25:19 dist-test-slave-qsm9 krb5kdc[23099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.253@KRBTEST.COM for kudu/127.0.116.252@KRBTEST.COM
I20260926 20:25:19.153244 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:19.143885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:37071 (local address 127.0.0.1:50964)
0926 20:25:19.144203 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:19.144216 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:25:19.144279 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:25:19.144750 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:25:19.144753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:25:19.144934 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:25:19.145025 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:19.145031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.146031 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.146034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:19.146838 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:19.146850 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.147666 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.147673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:19.147768 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:19.148200 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:25:19.148215 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:25:19.149463 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:25:19.151559 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:19.151564 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:19.151566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:19.151853 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:19.152166 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:19.152169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:19.152171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:19.152254 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:19.152696 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:25:19.152699 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:25:19.152792 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 20:25:19.152938 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 20:25:19.153079 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249}
I20260926 20:25:19.153584 23180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277: Neither blocks nor log segments found. Creating new log.
I20260926 20:25:19.153952 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:19.144419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50964 (local address 127.0.116.252:37071)
0926 20:25:19.144542 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.144546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.144557 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:25:19.144588 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.144591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.144629 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.145126 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.145202 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.145910 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.146994 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.147482 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.147943 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:19.149618 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:19.149622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:19.149625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:19.149657 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:19.151400 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:19.151977 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:19.151981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:19.151983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:19.152065 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:19.152364 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:19.152367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:19.152368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:19.152583 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:19.153569 (+ 986us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:19.153652 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 20:25:19.153775 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:25:19.154260 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:19.142376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57796 (local address 127.0.116.254:34085)
0926 20:25:19.142556 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.142559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.142568 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:25:19.142595 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.142597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.142626 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.142687 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.143477 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.144301 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.145491 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.145809 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.145885 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:19.150138 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:19.150142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:19.150145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:19.150178 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:19.152112 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:19.152643 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:19.152650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:19.152653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:19.152750 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:19.153082 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:19.153085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:19.153088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:19.153313 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:19.153480 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:19.153976 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 20:25:19.154109 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260926 20:25:19.154440 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:19.141606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48666 (local address 127.0.116.253:35117)
0926 20:25:19.141736 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.141741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.142078 (+ 337us) server_negotiation.cc:407] Connection header received
0926 20:25:19.142275 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.142278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.142342 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.142397 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.143133 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.143878 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.145560 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.145885 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.145957 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:19.150153 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:19.150157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:19.150159 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:19.150184 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:19.152453 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:19.152947 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:19.152951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:19.152953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:19.153037 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:19.153328 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:19.153334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:19.153339 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:19.153540 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:19.153623 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:19.154173 (+ 550us) server_negotiation.cc:299] Negotiation successful
0926 20:25:19.154291 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:25:19.154387 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:19.141518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.253:35117 (local address 127.0.0.1:48666)
0926 20:25:19.142002 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:19.142019 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:25:19.142150 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:25:19.142794 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:25:19.142796 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:25:19.142929 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:25:19.143015 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:19.143019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.144676 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.144679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:19.145408 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:19.145418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.146102 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.146106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:19.146201 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:19.146806 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:25:19.146820 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:25:19.149486 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:25:19.152574 (+ 3088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:19.152580 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:19.152582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:19.152820 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:19.153150 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:19.153153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:19.153155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:19.153227 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:19.153856 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:25:19.153862 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:25:19.153974 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 20:25:19.154111 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 20:25:19.154234 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399}
I20260926 20:25:19.155184 23180 log.cc:824] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277: Log is configured to *not* fsync() on all Append() calls
I20260926 20:25:19.155740 23331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:25:19.157244 23255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b76032762f484aa93ec99066eebe49: Bootstrap starting.
I20260926 20:25:19.160235 23255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65b76032762f484aa93ec99066eebe49: Neither blocks nor log segments found. Creating new log.
I20260926 20:25:19.161092 23255 log.cc:824] T 00000000000000000000000000000000 P 65b76032762f484aa93ec99066eebe49: Log is configured to *not* fsync() on all Append() calls
I20260926 20:25:19.161883 23180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277: No bootstrap required, opened a new log
Sep 26 20:25:19 dist-test-slave-qsm9 krb5kdc[23099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.116.253@KRBTEST.COM
I20260926 20:25:19.163152 23255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b76032762f484aa93ec99066eebe49: No bootstrap required, opened a new log
Sep 26 20:25:19 dist-test-slave-qsm9 krb5kdc[23099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790454319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.116.252@KRBTEST.COM for kudu/127.0.116.252@KRBTEST.COM
I20260926 20:25:19.166774 23180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816f68eed11341508a286efe1aaa7277" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34085 } } peers { permanent_uuid: "65b76032762f484aa93ec99066eebe49" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } } peers { permanent_uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } }
I20260926 20:25:19.167037 23180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816f68eed11341508a286efe1aaa7277, State: Initialized, Role: FOLLOWER
I20260926 20:25:19.166975 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:19.156419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48668 (local address 127.0.116.253:35117)
0926 20:25:19.156511 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.156515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.156527 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:25:19.156600 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.156602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.156634 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.156690 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.157460 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.158014 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.159068 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.159336 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.159421 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:19.162556 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:19.162560 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:19.162563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:19.162586 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:19.164732 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:19.165519 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:19.165523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:19.165525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:19.165608 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:19.165926 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:19.165929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:19.165931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:19.166142 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:19.166308 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:19.166675 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 20:25:19.166808 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 20:25:19.167415 23180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816f68eed11341508a286efe1aaa7277" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34085 } } peers { permanent_uuid: "65b76032762f484aa93ec99066eebe49" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } } peers { permanent_uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } }
I20260926 20:25:19.167680 23255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65b76032762f484aa93ec99066eebe49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816f68eed11341508a286efe1aaa7277" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34085 } } peers { permanent_uuid: "65b76032762f484aa93ec99066eebe49" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } } peers { permanent_uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } }
I20260926 20:25:19.167903 23255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65b76032762f484aa93ec99066eebe49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b76032762f484aa93ec99066eebe49, State: Initialized, Role: FOLLOWER
I20260926 20:25:19.167984 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:19.156309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50970 (local address 127.0.116.252:37071)
0926 20:25:19.159633 (+ 3324us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.159636 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.159646 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:25:19.159675 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.159677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.159732 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.159813 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.160278 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.160974 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.161984 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.162289 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.162356 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:19.164645 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:19.164650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:19.164653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:19.164673 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:19.166330 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:19.166769 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:19.166772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:19.166774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:19.166844 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:19.167099 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:19.167102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:19.167104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:19.167290 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:19.167356 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:19.167704 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 20:25:19.167820 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3251}
I20260926 20:25:19.168226 23255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65b76032762f484aa93ec99066eebe49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816f68eed11341508a286efe1aaa7277" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34085 } } peers { permanent_uuid: "65b76032762f484aa93ec99066eebe49" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } } peers { permanent_uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } }
I20260926 20:25:19.168391 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:19.156173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.116.252:37071 (local address 127.0.0.1:50970)
0926 20:25:19.156258 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 20:25:19.156272 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:25:19.156343 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:25:19.159835 (+ 3492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:25:19.159838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:25:19.160030 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:25:19.160152 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:19.160162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.161106 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.161112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:19.161864 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:25:19.161875 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.162486 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.162491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:25:19.162560 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:19.163025 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:25:19.163039 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:25:19.164514 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:25:19.166442 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:19.166444 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:19.166447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:19.166672 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:19.166931 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:25:19.166933 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:25:19.166935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:25:19.167013 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:25:19.167410 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:25:19.167413 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:25:19.167485 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 20:25:19.168129 (+ 644us) client_negotiation.cc:240] Negotiation successful
0926 20:25:19.168260 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 20:25:19.168697 23340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b76032762f484aa93ec99066eebe49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816f68eed11341508a286efe1aaa7277" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34085 } } peers { permanent_uuid: "65b76032762f484aa93ec99066eebe49" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } } peers { permanent_uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } } }
I20260926 20:25:19.168821 23340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b76032762f484aa93ec99066eebe49 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:25:19.169047 23180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:25:19.169454 23255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65b76032762f484aa93ec99066eebe49 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:25:19.171824 23331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64e8ba4c1cc649468b43e4e3caf0a6e1: Bootstrap starting.
I20260926 20:25:19.174542 23331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64e8ba4c1cc649468b43e4e3caf0a6e1: Neither blocks nor log segments found. Creating new log.
W20260926 20:25:19.174328 23362 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65b76032762f484aa93ec99066eebe49: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:25:19.176265 23362 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:25:19.176870 23339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816f68eed11341508a286efe1aaa7277" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34085 } } peers { permanent_uuid: "65b76032762f484aa93ec99066eebe49" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } } peers { permanent_uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } } }
I20260926 20:25:19.177119 23339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:25:19.183166 467 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:25:19.175431 23331 log.cc:824] T 00000000000000000000000000000000 P 64e8ba4c1cc649468b43e4e3caf0a6e1: Log is configured to *not* fsync() on all Append() calls
W20260926 20:25:19.174326 23356 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:25:19.184644 23356 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:25:19.187755 23154 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "816f68eed11341508a286efe1aaa7277"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57796
I20260926 20:25:19.188071 23154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 20:25:19.188185 23154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:25:19.189639 23154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816f68eed11341508a286efe1aaa7277" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34085 } } peers { permanent_uuid: "65b76032762f484aa93ec99066eebe49" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } } peers { permanent_uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } }
I20260926 20:25:19.187394 23187 master_runner.cc:186] Error getting master registration for 127.0.116.252:37071: OK, instance_id { permanent_uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1" instance_seqno: 1790454319113100 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:25:19.190747 23187 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:25:19.191818 23229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "816f68eed11341508a286efe1aaa7277" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "65b76032762f484aa93ec99066eebe49"
I20260926 20:25:19.192065 23229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65b76032762f484aa93ec99066eebe49 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:25:19.193380 23229 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65b76032762f484aa93ec99066eebe49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816f68eed11341508a286efe1aaa7277 in term 1.
I20260926 20:25:19.193912 23128 leader_election.cc:302] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b76032762f484aa93ec99066eebe49, 816f68eed11341508a286efe1aaa7277; no voters:
I20260926 20:25:19.191030 23112 master_runner.cc:186] Error getting master registration for 127.0.116.252:37071: OK, instance_id { permanent_uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1" instance_seqno: 1790454319113100 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:25:19.194278 23112 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:25:19.194399 23154 leader_election.cc:288] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277 [CANDIDATE]: Term 1 election: Requested vote from peers 65b76032762f484aa93ec99066eebe49 (127.0.116.253:35117), 64e8ba4c1cc649468b43e4e3caf0a6e1 (127.0.116.252:37071)
I20260926 20:25:19.196625 23331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64e8ba4c1cc649468b43e4e3caf0a6e1: No bootstrap required, opened a new log
I20260926 20:25:19.196645 23339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:25:19.197033 23339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277 [term 1 LEADER]: Becoming Leader. State: Replica: 816f68eed11341508a286efe1aaa7277, State: Running, Role: LEADER
I20260926 20:25:19.197614 23339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816f68eed11341508a286efe1aaa7277" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34085 } } peers { permanent_uuid: "65b76032762f484aa93ec99066eebe49" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } } peers { permanent_uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } }
I20260926 20:25:19.200771 23331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64e8ba4c1cc649468b43e4e3caf0a6e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816f68eed11341508a286efe1aaa7277" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34085 } } peers { permanent_uuid: "65b76032762f484aa93ec99066eebe49" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } } peers { permanent_uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } }
I20260926 20:25:19.201030 23331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64e8ba4c1cc649468b43e4e3caf0a6e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64e8ba4c1cc649468b43e4e3caf0a6e1, State: Initialized, Role: FOLLOWER
I20260926 20:25:19.201351 23331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64e8ba4c1cc649468b43e4e3caf0a6e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816f68eed11341508a286efe1aaa7277" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34085 } } peers { permanent_uuid: "65b76032762f484aa93ec99066eebe49" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } } peers { permanent_uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } }
I20260926 20:25:19.201776 23339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 816f68eed11341508a286efe1aaa7277. Latest consensus state: current_term: 1 leader_uuid: "816f68eed11341508a286efe1aaa7277" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816f68eed11341508a286efe1aaa7277" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34085 } } peers { permanent_uuid: "65b76032762f484aa93ec99066eebe49" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } } peers { permanent_uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } } }
I20260926 20:25:19.202378 23339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277 [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:19.202745 23331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64e8ba4c1cc649468b43e4e3caf0a6e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:25:19.202801 23305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "816f68eed11341508a286efe1aaa7277" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1"
I20260926 20:25:19.203030 23305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64e8ba4c1cc649468b43e4e3caf0a6e1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:25:19.204946 23305 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 64e8ba4c1cc649468b43e4e3caf0a6e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816f68eed11341508a286efe1aaa7277 in term 1.
I20260926 20:25:19.207697 23372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64e8ba4c1cc649468b43e4e3caf0a6e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816f68eed11341508a286efe1aaa7277" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34085 } } peers { permanent_uuid: "65b76032762f484aa93ec99066eebe49" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } } peers { permanent_uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } } }
I20260926 20:25:19.207840 23372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64e8ba4c1cc649468b43e4e3caf0a6e1 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:25:19.211833 23384 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64e8ba4c1cc649468b43e4e3caf0a6e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:25:19.211973 23384 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:25:19.220679 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:19.201482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57800 (local address 127.0.116.254:34085)
0926 20:25:19.201613 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.201618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.208406 (+ 6788us) server_negotiation.cc:407] Connection header received
0926 20:25:19.208448 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.208451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.208499 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.208585 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.212380 (+ 3795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.213184 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.214411 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.214762 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.214838 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:19.216883 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:19.216888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:19.216890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:19.216920 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:19.218675 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:19.219192 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:19.219196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:19.219198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:19.219286 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:19.219637 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:19.219641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:19.219643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:19.219871 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:19.219994 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:19.220346 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 20:25:19.220490 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:25:19.221182 23385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:25:19.231009 23385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:25:19.246403 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:19.206406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.0.116.252:37071)
0926 20:25:19.206514 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.206520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.223131 (+ 16611us) server_negotiation.cc:407] Connection header received
0926 20:25:19.223407 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.223410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.223466 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.223561 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.227752 (+ 4191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.228560 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.235718 (+ 7158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.236084 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.236168 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:19.240402 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:19.240407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:19.240410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:19.240444 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:19.242110 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:19.242627 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:19.242632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:19.242633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:19.242722 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:19.244981 (+ 2259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:19.244984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:19.244986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:19.245213 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:19.245343 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:19.246076 (+ 733us) server_negotiation.cc:299] Negotiation successful
0926 20:25:19.246233 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:25:19.247822 23305 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 64e8ba4c1cc649468b43e4e3caf0a6e1 [term 1 FOLLOWER]: Refusing update from remote peer 816f68eed11341508a286efe1aaa7277: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:25:19.248644 23339 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:25:19.249243 23229 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65b76032762f484aa93ec99066eebe49 [term 1 FOLLOWER]: Refusing update from remote peer 816f68eed11341508a286efe1aaa7277: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:25:19.250037 23339 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b76032762f484aa93ec99066eebe49" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:25:19.254091 23340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b76032762f484aa93ec99066eebe49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 816f68eed11341508a286efe1aaa7277. Latest consensus state: current_term: 1 leader_uuid: "816f68eed11341508a286efe1aaa7277" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816f68eed11341508a286efe1aaa7277" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34085 } } peers { permanent_uuid: "65b76032762f484aa93ec99066eebe49" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } } peers { permanent_uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } } }
I20260926 20:25:19.254233 23340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b76032762f484aa93ec99066eebe49 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:25:19.254412 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:19.206052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48674 (local address 127.0.116.253:35117)
0926 20:25:19.207415 (+ 1363us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.207420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.224969 (+ 17549us) server_negotiation.cc:407] Connection header received
0926 20:25:19.225698 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.225702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.225760 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.225814 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.233769 (+ 7955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.234552 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.238204 (+ 3652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:19.238560 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:19.238637 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:19.243060 (+ 4423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:25:19.243065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:25:19.243068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:25:19.243098 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:25:19.244833 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:19.248134 (+ 3301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:19.248138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:19.248140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:19.248240 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:25:19.253367 (+ 5127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:25:19.253371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:25:19.253372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:25:19.253614 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:25:19.253736 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:19.254114 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 20:25:19.254248 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1277}
I20260926 20:25:19.254897 23372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64e8ba4c1cc649468b43e4e3caf0a6e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 816f68eed11341508a286efe1aaa7277. Latest consensus state: current_term: 1 leader_uuid: "816f68eed11341508a286efe1aaa7277" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816f68eed11341508a286efe1aaa7277" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34085 } } peers { permanent_uuid: "65b76032762f484aa93ec99066eebe49" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } } peers { permanent_uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } } }
I20260926 20:25:19.255050 23372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64e8ba4c1cc649468b43e4e3caf0a6e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:25:19.259857 23340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b76032762f484aa93ec99066eebe49 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "816f68eed11341508a286efe1aaa7277" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816f68eed11341508a286efe1aaa7277" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34085 } } peers { permanent_uuid: "65b76032762f484aa93ec99066eebe49" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } } peers { permanent_uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } } }
I20260926 20:25:19.259985 23340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b76032762f484aa93ec99066eebe49 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:25:19.260617 23389 mvcc.cc:205] Tried to move back new op lower bound from 7333700891614117888 to 7333700891439669248. Current Snapshot: MvccSnapshot[applied={T|T < 7333700891614117888}]
I20260926 20:25:19.259402 23339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "816f68eed11341508a286efe1aaa7277" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816f68eed11341508a286efe1aaa7277" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34085 } } peers { permanent_uuid: "65b76032762f484aa93ec99066eebe49" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } } peers { permanent_uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } } }
I20260926 20:25:19.261843 23339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277 [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:19.263645 23392 mvcc.cc:205] Tried to move back new op lower bound from 7333700891614117888 to 7333700891439669248. Current Snapshot: MvccSnapshot[applied={T|T < 7333700891614117888}]
I20260926 20:25:19.273403 23339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "816f68eed11341508a286efe1aaa7277" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816f68eed11341508a286efe1aaa7277" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34085 } } peers { permanent_uuid: "65b76032762f484aa93ec99066eebe49" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } } peers { permanent_uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } } }
I20260926 20:25:19.273558 23339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277 [sys.catalog]: This master's current role is: LEADER
I20260926 20:25:19.274664 23372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64e8ba4c1cc649468b43e4e3caf0a6e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "816f68eed11341508a286efe1aaa7277" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816f68eed11341508a286efe1aaa7277" member_type: VOTER last_known_addr { host: "127.0.116.254" port: 34085 } } peers { permanent_uuid: "65b76032762f484aa93ec99066eebe49" member_type: VOTER last_known_addr { host: "127.0.116.253" port: 35117 } } peers { permanent_uuid: "64e8ba4c1cc649468b43e4e3caf0a6e1" member_type: VOTER last_known_addr { host: "127.0.116.252" port: 37071 } } }
I20260926 20:25:19.274817 23372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64e8ba4c1cc649468b43e4e3caf0a6e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:25:19.275393 23390 mvcc.cc:205] Tried to move back new op lower bound from 7333700891614117888 to 7333700891439669248. Current Snapshot: MvccSnapshot[applied={T|T < 7333700891614117888}]
I20260926 20:25:19.275650 23385 catalog_manager.cc:1347] Generated new cluster ID: b099a05cd4e047849255fd09fb68ef71
I20260926 20:25:19.275712 23385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:25:19.311472 23385 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:25:19.312881 23385 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:25:19.332352 23385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 816f68eed11341508a286efe1aaa7277: Generated new TSK 0
I20260926 20:25:19.333251 23385 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:25:19.374434 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:19.370873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57808 (local address 127.0.116.254:34085)
0926 20:25:19.371023 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.371028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.373151 (+ 2123us) server_negotiation.cc:407] Connection header received
0926 20:25:19.373409 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.373413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.373474 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.373535 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.374258 (+ 723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57808: BlockingRecv error: recv got EOF from 127.0.0.1:57808 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 20:25:19.376456 23336 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:19.371743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50992 (local address 127.0.116.252:37071)
0926 20:25:19.371868 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.371873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.374985 (+ 3112us) server_negotiation.cc:407] Connection header received
0926 20:25:19.375434 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.375438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.375562 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.375622 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.376307 (+ 685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50992: BlockingRecv error: recv got EOF from 127.0.0.1:50992 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 20:25:19.377777 23261 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:19.372478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48678 (local address 127.0.116.253:35117)
0926 20:25:19.372597 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.372602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.375321 (+ 2719us) server_negotiation.cc:407] Connection header received
0926 20:25:19.376844 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.376848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.376904 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.376961 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.377657 (+ 696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48678: BlockingRecv error: recv got EOF from 127.0.0.1:48678 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 20:25:19.383255 23261 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:19.381383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48682 (local address 127.0.116.253:35117)
0926 20:25:19.381496 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.381501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.382143 (+ 642us) server_negotiation.cc:407] Connection header received
0926 20:25:19.382410 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.382413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.382498 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.382559 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.383137 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48682: BlockingRecv error: recv got EOF from 127.0.0.1:48682 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 20:25:19.388670 23261 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:19.387057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48696 (local address 127.0.116.253:35117)
0926 20:25:19.387166 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.387170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.387712 (+ 542us) server_negotiation.cc:407] Connection header received
0926 20:25:19.387935 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.387939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.387991 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.388044 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.388552 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48696: BlockingRecv error: recv got EOF from 127.0.0.1:48696 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 20:25:19.398763 23261 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:19.392982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48704 (local address 127.0.116.253:35117)
0926 20:25:19.393103 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.393108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.397749 (+ 4641us) server_negotiation.cc:407] Connection header received
0926 20:25:19.397974 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.397979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.398044 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.398097 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.398639 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48704: BlockingRecv error: recv got EOF from 127.0.0.1:48704 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 20:25:19.407760 23261 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:19.406401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48710 (local address 127.0.116.253:35117)
0926 20:25:19.406547 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.406552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.406802 (+ 250us) server_negotiation.cc:407] Connection header received
0926 20:25:19.406984 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.406987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.407043 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.407142 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.407640 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48710: BlockingRecv error: recv got EOF from 127.0.0.1:48710 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 20:25:19.419378 23261 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:19.417972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48720 (local address 127.0.116.253:35117)
0926 20:25:19.418129 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.418136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.418348 (+ 212us) server_negotiation.cc:407] Connection header received
0926 20:25:19.418597 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.418600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.418649 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.418735 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.419238 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48720: BlockingRecv error: recv got EOF from 127.0.0.1:48720 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 20:25:19.438845 23261 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:19.437662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48736 (local address 127.0.116.253:35117)
0926 20:25:19.437904 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.437908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.437928 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:25:19.438073 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.438076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.438138 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.438216 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.438710 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48736: BlockingRecv error: recv got EOF from 127.0.0.1:48736 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260926 20:25:19.474102 23261 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:19.473001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48748 (local address 127.0.116.253:35117)
0926 20:25:19.473171 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.473176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.473257 (+ 81us) server_negotiation.cc:407] Connection header received
0926 20:25:19.473382 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.473384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.473433 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.473511 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.473952 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48748: BlockingRecv error: recv got EOF from 127.0.0.1:48748 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 20:25:19.541541 23261 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:19.540453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48764 (local address 127.0.116.253:35117)
0926 20:25:19.540610 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.540614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.540691 (+ 77us) server_negotiation.cc:407] Connection header received
0926 20:25:19.540812 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.540815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.540864 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.540938 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.541399 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48764: BlockingRecv error: recv got EOF from 127.0.0.1:48764 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 20:25:19.673192 23261 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:19.671889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48780 (local address 127.0.116.253:35117)
0926 20:25:19.672010 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.672056 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.672291 (+ 235us) server_negotiation.cc:407] Connection header received
0926 20:25:19.672456 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.672464 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.672516 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.672606 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.673057 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48780: BlockingRecv error: recv got EOF from 127.0.0.1:48780 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 20:25:19.932579 23261 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:25:19.931445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48796 (local address 127.0.116.253:35117)
0926 20:25:19.931654 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:19.931657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:19.931675 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:25:19.931816 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:19.931819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:19.931868 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:19.931961 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:25:19.932451 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48796: BlockingRecv error: recv got EOF from 127.0.0.1:48796 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 20:25:20.178009 23362 catalog_manager.cc:1259] Loaded cluster ID: b099a05cd4e047849255fd09fb68ef71
I20260926 20:25:20.178102 23362 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 65b76032762f484aa93ec99066eebe49: loading cluster ID for follower catalog manager: success
I20260926 20:25:20.179805 23362 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 65b76032762f484aa93ec99066eebe49: acquiring CA information for follower catalog manager: success
I20260926 20:25:20.181034 23362 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 65b76032762f484aa93ec99066eebe49: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:25:20.213300 23384 catalog_manager.cc:1259] Loaded cluster ID: b099a05cd4e047849255fd09fb68ef71
I20260926 20:25:20.213414 23384 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 64e8ba4c1cc649468b43e4e3caf0a6e1: loading cluster ID for follower catalog manager: success
I20260926 20:25:20.215137 23384 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 64e8ba4c1cc649468b43e4e3caf0a6e1: acquiring CA information for follower catalog manager: success
I20260926 20:25:20.216383 23384 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 64e8ba4c1cc649468b43e4e3caf0a6e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:25:20.451066 23461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:20.446901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48802 (local address 127.0.116.253:35117)
0926 20:25:20.447183 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:20.447187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:20.447204 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:25:20.447332 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:20.447334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:20.447572 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:20.447700 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:25:20.448052 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:20.448594 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:20.449474 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:20.449723 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:20.449840 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:20.450149 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:25:20.450667 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:25:20.450813 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:20.450855 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 20:25:20.450917 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":99,"threads_started":1}
I20260926 20:25:20.458613 23467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:20.454538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51002 (local address 127.0.116.252:37071)
0926 20:25:20.454809 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:20.454814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:20.454830 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:25:20.455046 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:20.455049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:20.455187 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:20.455319 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:25:20.455583 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:20.456103 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:20.457000 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:20.457247 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:20.457363 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:20.457685 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:25:20.458165 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:25:20.458311 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:20.458377 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 20:25:20.458438 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
I20260926 20:25:20.467310 23467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:20.462940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51018 (local address 127.0.116.252:37071)
0926 20:25:20.463077 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:20.463083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:20.463097 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:25:20.463143 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:20.463146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:20.463191 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:20.463271 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:25:20.463575 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:20.464364 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:20.465847 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:20.466292 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:20.466383 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:20.466732 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:25:20.466820 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:25:20.466911 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:20.467046 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 20:25:20.467115 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:25:20.467643 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:20.462644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57822 (local address 127.0.116.254:34085)
0926 20:25:20.463031 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:20.463038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:20.463053 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:25:20.463153 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:20.463158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:20.463355 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:20.463527 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:25:20.463757 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:20.464287 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:20.465435 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:20.465686 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:20.465980 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:20.466415 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:25:20.466725 (+ 310us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:25:20.466843 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:20.467314 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 20:25:20.467388 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":82,"threads_started":1}
I20260926 20:25:20.467818 23461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:25:20.462435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48812 (local address 127.0.116.253:35117)
0926 20:25:20.462850 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0926 20:25:20.462858 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:25:20.462875 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:25:20.462932 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:25:20.462934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:25:20.462990 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:25:20.463064 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:25:20.463404 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:20.464292 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:20.465894 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:25:20.466193 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:25:20.466324 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:25:20.467261 (+ 937us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:25:20.467347 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:25:20.467395 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 20:25:20.467612 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 20:25:20.467681 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314}
I20260926 20:25:20.478905 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 23112
I20260926 20:25:20.485239 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 23187
I20260926 20:25:20.491158 467 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3m9pMa/build/debug/bin/kudu with pid 23262
2026-09-26T20:25:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3222 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6358 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (286510 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST